airesearch/wangchanberta-base-wiki-spm

The architecture of the pretrained model is based on RoBERTa [Liu et al., 2019]. The model was trained on 32 V100 GPUs for 31,250 steps with the batch size of 8,192 (16 sequences per device with 16 sequences) and a sequence length of 512 tokens . We use the Wikipedia dump articles on 20 August 2020 (dumps.wikimedia.org/thwiki/20200820/) We opt out lists, and tables . The getting started notebook of WangchanBERTa model can be found at this Colab notebook . The training data includes 944,782 sentences for training set, 24,863 sentences for validation set, and 24,,