NLP4H/ms_bert

MS-BERT was pre-trained on notes from neurological examination for Multiple Sclerosis patients at St. Michael's Hospital in Toronto, Canada . The dataset contained approximately 75,000 clinical notes, for about 5000 patients, totaling to over 35.7 million words . The gender split within the dataset was observed to be 72% female and 28% male (which reflects the natural discrepancy seen in MS). The results are not intended for direct diagnostic use or medical decision-making without review and oversight by a clinical professional . We further pre-train it using the masked language modelling task from the huggingface transformers library . The hyperparameters can be found in the config file in this repository or here,