loodos/electra-base-turkish-uncased-discriminator

Huggingface's tokenizers have a bug concerning letters "ı, i, I, İ" and non-ASCII Turkish specific letters . NFD/NFKD normalization is not proper for Turkish . Until it is fixed, Python's default string.lower() and string.upper() make the conversions "I" and "İ" to 'i' or 'I'respectively. However, in Turkish, 'I and 'İ' are two different letters. This causes wrong tokenization, wrong training and loss of information. Until it's fixed, in case you want to train your model with uncased data, we provide a simple text normalization module,