kuisailab/albert-xlarge-arabic

The models were pretrained on 4.4 Billion words: Arabic version of OSCAR (unshuffled version of the corpus) - filtered from Common Crawl (recent dump of Arabic Wikipedia) The final version of corpus contains some non-Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER . The models are not restricted to Modern Standard Arabic, they contain some dialectical Arabic too . They were trained using Google ALBERT's GitHub repository on a single TPU v3-8 provided for free from TFRC . The training procedure follows training settings of bert with some changes: trained for 7M training steps with batch,