moussaKam/mbarthez

A corpus of 66GB of french raw text is used to carry out the pretraining . A multilingual BART mBART boosted its performance in both discriminative and generative tasks . We call the french adapted version mBARThez. It is pretrained by learning to reconstruct a corrupted input sentence . The new version of BART is based on a BERT-based French language model such as CamemBERT and FlauBERT, BARThez is particularly well-suited for . generative . tasks (such as abstractive summarization), since not only its encoder but also its decoder is . pretrained from scratch, we continue to . pretraining of,