distilroberta-base

DistilRoBERTa is a distilled version of the RoBERTa-base model . It follows the same training procedure as DistilBERT . The model has 6 layers, 768 dimension and 12 heads, totalizing 82M parameters . The code for the distillation process can be found here . It was pre-trained on OpenWebTextCorpus, a reproduction of OpenAI's WebText dataset (it is ~4 times less training data than the teacher RoberTa) The model is case-sensitive: it makes a difference between English and English . On average it is twice as fast as Roberta-base. The model achieves the following results:. When fine-tuned,