allegro/herbert-klej-cased-tokenizer-v1

HerBERT tokenizer is a character level byte-pair encoding with a vocabulary size of 50k tokens . The tokenizer was trained on Wolne Lektury and a subset of the National Corpus of Polish with fastBPE library .Tokenizer utilize XLMTokenizer implementation from transformers. It is created by Allegro Machine Learning Research team. If you use this tokenizer, please cite the following paper: KLEJ: Comprehensive Benchmark for Polish Language Understanding. The paper is accepted at ACL 2020, as soon as proceedings appear, we will update the BibTeX. We will update BibTeX to provide an updated version of the Bibtex. We are happy to provide a,