distilgpt2

DistilGPT2 pretrained with supervision of GPT2 on OpenWebTextCorpus, a reproduction of OpenAI's WebText dataset . The model has 6 layers, 768 dimension and 12 heads, totalizing 82M parameters (compared to 124M parameters for Gpt2) On average, DistilGP2 is two times faster than GPT1 . On the WikiText-103 benchmark, GPT 2 reaches a perplexity on the test set of 16.3 compared to 21.1 for DistilPG2 (after fine-tuning on the train set) After fine tuning on the . train set, the . test set . was found to be two times,