microsoft/deberta-base

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder . With those two improvements, DeberTa out performs RoberTa on a majority of NLU tasks with 80GB training data . We present the dev results on SQuAD 1.1/2.0 and MNLI tasks. Please cite the paper: DeBERTA: Decoding-enhanced BERT with Disentangled Attention. [DeBERTa] for more details and updates. Please check the official repository for more information and updates from DeBERta. It is available to download and use in the Open Reviewer's Open Reviewers' Guide to the,