allenai/biomed_roberta_base

We adapt RoBERTa-base to 2.68 million scientific papers from the Semantic Scholar corpus via continued pretraining . This amounts to 7.55B tokens and 47GB of data . We use the full text of the papers in training, not just abstracts . Specific details of the adaptive pretraining procedure can be found in Gururangan et. al, 2020 . The model achieves competitive performance to state of the art models on a number of NLP tasks in the biomedical domain (numbers are mean (standard deviation) over 3+ random seeds) If using this model, please cite the following paper: The Proceedings of ACL, or the authors of this paper: {,