DeepPavlov/bert-base-bg-cs-pl-ru-cased

SlavicBERT was trained on Russian News and four Wikipedias: Bulgarian, Czech, Polish, Russian and Russian . Subtoken vocabulary was built using this data . Multilingual BERT was used as an initialization for SlavicberT . Slavic (bg, cs, pl, ru), cased, 12-layer, 768‑hidden, 12‑heads, 180M parameters were used to train the new system . The data was used to build a subtoken vocabulary for the new language . The new language was created using the data from BERT and the language-recognition algorithm . The code is based on the language of Slavic, Czech and Russian news and Wikipedia articles .,