loodos/electra-small-turkish-uncased-discriminator

Huggingface's tokenizers (which were written in Python) have a bug concerning letters "ı, i, I, İ" and non-ASCII Turkish specific letters . NFD/NFKD normalization is not proper for Turkish . Python's default string.lower() and string.upper() make the conversions to 'I' and 'İ' to 'i' The bug causes wrong tokenization, wrong training and loss of information. Until it is fixed, we provide a simple text normalization module (TextNormalization() in the code snippet above) in our repo . We release cased and uncased versions of most recent language models for Turkish at Lood,