airesearch/wangchanberta-base-wiki-sefr

The architecture of the pretrained model is based on RoBERTa [Liu et al., 2019]. The model was trained on 32 V100 GPUs for 31,250 steps with the batch size of 8,192 (16 sequences per device with 16 accumulation steps) and a sequence length of 512 tokens . The getting started notebook of WangchanBERTa model can be found at this Colab notebook . The total number of word-level tokens in the vocabulary is 92,177 . We sample sentences contigously to have the length of at most 512 tokens. For some sentences that overlap the boundary of the 512 tokens, we split such sentence with an additional token as document separator . We opt,