microsoft/unihanlm-base

Chinese and Japanese share many characters with similar surface morphology . We propose a self-supervised Chinese-Japanese pretrained masked language model (MLM) with a novel two-stage coarse-to-fine training approach . The training corpus is solely from Wikipedia so the model may perform worse on informal text data . The proposed UnihanLM is effective on both mono- and cross-lingual tasks, shedding light on a new path to exploit the homology of languages. The training data is only from Wikipedia . Be careful with English words! The tokenizer would cut it to characters . The model performs better on a variety of Chinese and . Japanese NLP benchmarks, showing that our proposed Un,