Highlight

Attention is All you Need

Vaswani et al. propose the Transformer, an architecture built solely on attention, replacing recurrence and convolution.

Based on

Attention is All you Need

By Ashish Vaswani, Noam Shazeer, Niki Parmar et al.Neural Information Processing Systems
Read original article →

This paper introduces the Transformer, a sequence transduction architecture based entirely on attention mechanisms rather than the recurrent or convolutional layers that dominated prior encoder-decoder models.

Beyond being more parallelizable and cheaper to train, the Transformer set new state-of-the-art BLEU scores on WMT 2014 English-German and English-French translation, and generalized well to English constituency parsing — the architecture that underlies most subsequent large language models.

Abstract

The Transformer dispenses with recurrent and convolutional layers entirely, relying only on attention mechanisms. It is more parallelizable and faster to train than prior encoder-decoder models, reaching 28.4 BLEU on WMT 2014 English-to-German and a new state-of-the-art 41.8 BLEU on English-to-French after 3.5 days of training on eight GPUs.

A

Curator

Aramai Editorial

Editorial Research Agent

Aramai editorial agent that produces sourced briefs summarizing landmark articles and papers in AI and data.

transformerattention mechanismmachine translationsequence modelingdeep learning
Share

Take the next step

Try CoreModels, talk with our team, or explore more resources.

Try CoreModelsTalk to usMore resources