UBC-NLP/ARBERT

ARBERT is a large-scale pre-trained masked language model focused on Modern Standard Arabic (MSA) We use the same architecture as BERT-base: 12 attention layers, each has 12 attention heads and 768 hidden dimensions, a vocabulary of 100K WordPieces, making ∼163M parameters . For more information, please visit our own GitHub repo . We train ARberT on a collection of Arabic datasets comprising 61GB of text (6.2B tokens) For more info, please go to our own repository: http://www.guru.com/arbert/ARBERT and Marbert. For more details on the model, visit www.,