microsoft/MiniLM-L12-H384-uncased

MiniLM is a distilled model from the paper "MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers" Please note: This checkpoint can be an inplace substitution for BERT and it needs to be fine-tuned before use! We present the dev results on SQuAD 2.0 and several GLUE benchmark tasks. The full details of the MiniLM can be found in the original MiniLM repository. The model is distilled from an in-house pre-trained UniLM v2 model in BERT-Base size. It is 2.7x faster than BERT but fine-tuning on NLU,