asafaya/bert-base-arabic

Arabic BERT model was pretrained on 8.2 Billion words: Arabic version of OSCAR - filtered from Common Crawl . The model was trained using Google BERT's Github repository on a single TPU v3-8 provided for free from TFRC . The final version of corpus contains some non-Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER. The model is not restricted to Modern Standard Arabic, they contain some dialectical Arabic too. It was trained for 3M training steps with batchsize of 128, instead of 1M with batch size of 256. It can be installed using torch or tensorflow,