allegro/herbert-klej-cased-v1

HerBERT is a BERT-based Language Model trained on Polish Corpora with only MLM objective with dynamic masking of whole words . The training dataset was tokenized into subwords using a character level byte-pair encoding with a vocabulary size of 50k tokens . The tokenizer itself was trained on Wolne Lektury and a subset of National Corpus of Polish with a fastBPE library . The model was evaluated on the KLEJ benchmark, publicly available set of nine evaluation tasks for the Polish language understanding . It had the best average performance and obtained the best results for three of them . It can also be trained using AutoTokenizer and AutoModel . The data,