iarfmoose/roberta-small-bulgarian

This is a smaller version of RoBERTa-base-bulgarian with only 6 hidden layers . The model can be used for cloze tasks (masked language modeling) or finetuned on other tasks in Bulgarian . The training data is unfiltered text from the internet and may contain all sorts of biases . The batch size was limited to 8 due to GPU memory limitations . The data was trained on the following data: Newscrawl 1 million sentences 2017 from Leipzig Corpora Collection and Wikipedia . The dataset was trained for 160k steps and was trained using a masked language-modeling objective with dynamic masking as described here. The model was pretrained using a mask,