cl-tohoku/bert-large-japanese

This is a BERT model pretrained on texts in the Japanese language . It processes input texts with word-level tokenization based on the Unidic 2.1.2 dictionary (available in unidic-lite) and WordPiece subword tokenization . The model is trained with whole word masking enabled for the masked language modeling (MLM) objective . The models are trained on the Japanese version of Wikipedia . They are distributed under the terms of the Creative Commons Attribution-ShareAlike 3.0.0 . The training took about 5 days to finish. The model architecture is the same as the original BERT large model; 24 layers, 1024 dimensions of hidden states,