Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Transformer-XL is a neural architecture for language modeling that overcomes the fixed-length context limitation of standard Transformers. It introduces two ingredients: a segment-level recurrence mechanism that carries information across segments, and a novel positional encoding scheme. Together these let the model learn dependencies well beyond a fixed length without disrupting temporal coherence, while also resolving the context fragmentation problem.

As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, performs better on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. It improved state-of-the-art results across enwiki8, text8, WikiText-103, One Billion Word, and Penn Treebank, and when trained only on WikiText-103 could generate coherent, novel articles of thousands of tokens; code and pretrained models were released.