airesearch/wangchanberta-base-att-spm-uncased

The WangchanBERTa model was pretrained on assorted Thai text dataset . The total size of uncompressed text is 78.5GB . We also provide models for multiclass/multilabel text classification and token classification task . The script and documentation can be found at this reposiryory. The architecture of the pretrained model is based on RoBERTa [Liu et al., 2019]. The training data set is 38,034,638 unique Thai sentences with sequence length of 5 to 3 to 3.38 unique words . We have a pre-processed pre-processing and deduplication tool . We use the training data to train SentencePiece unig,