allenai/scibert_scivocab_cased

The training corpus was papers taken from Semantic Scholar . Corpus size is 1.14M papers, 3.1B tokens . We use the full text of the papers in training, not just abstracts . SciBERT has its own wordpiece vocabulary (scivocab) that's built to best match the training corpus . We trained cased and uncased versions of the models . The original repo can be found in the original repo of the original repository here . If using these models, please cite the following paper: "SciBERT: A Pretrained Language Model for Scientific Text", "D19-1371" by Iz Beltagy, Iz Lo, Kyle Cohan,