hfl/chinese-macbert-base

This repository contains the resources in our paper "Revisiting Pre-trained Models for Chinese Natural Language Processing" The paper will be published in "Findings of EMNLP" You can read our camera-ready paper through ACL Anthology or arXiv pre-print . We also incorporate the following techniques: Whole Word Masking (WWM), Sentence-Order Prediction (SOP) and MacBERT with novel MLM as correction pre-training task, which mitigates the discrepancy of pre- training and fine-tuning . We use a language model to predict the probability of the next word. We also use a text system to ca . ca .ulate the po,