iarfmoose/roberta-base-bulgarian

This is a version of RoBERTa-base pretrained on Bulgarian text . The model can be used for cloze tasks (masked language modeling) or finetuned on other tasks in Bulgarian . The training data is unfiltered text from the internet and may contain all sorts of biases. The batch size was limited to 8 due to GPU memory limitations. The model was trained on the following data: Newscrawl 1 million sentences 2017 from Leipzig Corpora Collection. It was trained for 200k steps. The data was trained using a masked language-modeling objective with dynamic masking as described here in this article . It was training data from OSCAR and Wikipedia .,