cambridgeltl/BioRedditBERT-uncased

We use the same pre-training script in the original google-research/bert repo . The model is initialised with BioBERT-Base v1.0 + PubMed 200K + PMC 270K . We train with a batch size of 64, a max sequence length of 64 and a learning rate of 2e-5 for 100k steps on two GeForce GTX 1080Ti (11 GB) GPUs . We follow the same 10-fold cross-validation procedure for all models and report the average result without fine-tuning . We also demonstrate results on a medical entity linking dataset also in the social media domain: AskAPatient (Limsopatham and Collier 2016,