Longformer: The Long-Document Transformer

The paper addresses the inability of transformer models to handle long sequences, which stems from self-attention scaling quadratically with sequence length. Longformer replaces standard self-attention with a mechanism that scales linearly, combining a local windowed attention with a task-motivated global attention, and is designed as a drop-in replacement so it can process documents of thousands of tokens or longer.

Longformer achieves state-of-the-art results on the text8 and enwik8 character-level language modeling benchmarks, and unlike much prior long-sequence work it is also pretrained and finetuned on downstream tasks, where it consistently outperforms RoBERTa and sets new state-of-the-art results on WikiHop and TriviaQA. The authors further introduce the Longformer-Encoder-Decoder (LED) for long-document sequence-to-sequence generation, demonstrating its effectiveness on arXiv summarization.