canwenxu/BERT-of-Theseus-MNLI

BERT-of-Theseus is a new compressed BERT by progressively replacing the components of the original BERT . We provide a 6-layer pretrained model on MNLI as a general-purpose model, which can transfer to other sentence classification tasks, outperforming DistillBERT on six tasks of GLUE (dev set) Please fine-tune it before use before use (like DistilBERT). The model is for Intermediate-Task Transfer Learning so it does not include the classification head for MNLI! Please fine tune it with the correct language for other tasks, such as MNLI or DistillBerT . The model can be used for Intermediate Task Transfer Learning .,