w11wo/javanese-bert-small

The Javanese BERT Small is a masked language model based on the BERT model . It was trained on the latest (late December 2020) Wikipedia articles . It achieved a perplexity of 22.00 on the validation dataset (20% of the articles) Many of the techniques used are based on a Hugging Face tutorial notebook written by Sylvain Gugger and fine-tuning tutorial written by Pierre Guillou . PyTorch was used as the backend framework during training, but the model remains compatible with TensorFlow . The model was trained for 5 epochs and the following is the final result once the training ended . All computation and development are done on Google Colabor,