onlplab/alephbert-base

Based on Google's BERT architecture (Devlin et al. 2018) Hebrew section (10 GB text, 20 million sentences). Wikipedia dump of Wikipedia (650 MB text, 3 million sentences) Tweets collected from the Twitter sample stream (7 GB text) Trained on a DGX machine (8 V100 GPUs) using the standard huggingface training procedure . Each section was first trained for 5 epochs with an initial learning rate set to 1e-4 . Then each section was trained for another 5 . epochs, for a total of 10 epochs . Total training time was 8 days, training took an average of 8 days . To optimize training time we split the data into 4,