microsoft/xtremedistil-l6-h384-uncased

The XtremeDistil is a distilled task-agnostic transformer model leveraging multi-task distillation techniques . The model has 6 layers, 384 hidden size, 12 attention heads corresponds to 22 million parameters with 5.3x speedup over BERT-base . The following table shows the results on GLUE dev set and SQuAD-v2 . The results are based on the following "Github" code: l6-h384 checkpoint with 6 layers . The code has been developed by the Association for Computational Linguistics (ACCL) and "MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained,