monsoon-nlp/dv-muril

This is an experiment in transfer learning, to insert Dhivehi word and word-piece tokens into Google's MuRIL model . This BERT-based model currently performs better than dv-wave ELECTRA on the Maldivian News Classification task . The model is similar to mBERT with no Thaana vocabulary, attach 1,100 words to Malayalam or English embeddings. Add remaining words from BertWordPieceTokenizer / vocab.txt to Bert word-word-piece tokenizer . BERT pretraining step: 89.8% (10k vocab): 90.7% (30k Vocab) before BERT training step:,