allenai/longformer-base-4096

Longformer is a transformer model for long documents . It supports sequences of length up to 4,096 . Longformer uses a combination of a sliding window (local) attention and global attention . The model is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. The Allen Institute for Artificial Intelligence (AI2) is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering . The project is an open-source project developed by the Allen Institute of Artificial Intelligence . It was developed in 2004 by Iz Beltagy and Matthew E. Peters and Arman Cohan . The paper has been published,