juliawabant/camembert-long

This model is an "extension" of [CamemBERT-base model] (https://camembert-model.fr/) to support up to 4096 tokens . The architecture was previously extended using AllenAI's code for the procedure descriped in the Longformer paper . The model can be pretained using FP16, or using TPUs with TensorFlow or PyTorch-XLA . It is Apache-2.0 licensed and can be pre-trained with FP16 or using a HuggingFace pipeline (fillmask) The Longformer is an extension of the RoBERTa model and CamemberT use the same architecture as RoberTa .,