huawei-noah/TinyBERT_General_4L_312D

TinyBERT is 7.5x smaller and 9.4x faster on inference than BERT-base . It performs a novel transformer distillation at both the pre-training and task-specific learning stages . In general distillation, we use the original Bert-base without fine-tuning as the teacher and a large-scale text corpus as the learning data . For more details about the techniques, refer to our paper:TinyBERT: Distilling BERT for Natural Language Understanding. The paper is published by Jiao, Xiaoqi and Yin, Yichun and Shang, Lifeng and Jiang, Xin and Chen, Xiao and Li, Linlin and Wang,,