EMBEDDIA/sloberta

Slovene model is a monolingual Slovene BERT-like model . It is closely related to French Camembert model . The corpora used for training the model have 3.47 billion tokens in total . The subword vocabulary contains 32,000 tokens . The scripts and programs used for data preparation and training are available on https://://github.com/clarinsi/Slovene-BERT-ToolSloBERTa . The model was trained for 200,000 iterations or about 98 epochs . The following corpora were used to train the model: Gigafida 2.0, Kas 1.0 and Janes-1.0 (only,