google/bigbird-roberta-large

BigBird relies on block sparse attention instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT . It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts . The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team . It is a pretrained model on English language using a masked language modeling (MLM) objective . The model is warm started from RoBERTa’s checkpoint. It used same sentencepiece vocabulary as,