nielsr/coref-bert-base

CorefBERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion . It was pretrained with two objectives: Masked Language Modeling (MLM) and Mention Reference Prediction (MRP) objectives . This is different from traditional neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like the future tokens . It allows the model to learn a bidirectional representation of the sentence . If you have a dataset of labeled sentences for instance, you can train a standard classifier using the features produced by the model as inputs . This way, the model learns an inner representation,