Highlight

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin et al. present BERT, a bidirectional Transformer pretraining method that set new state-of-the-art results on eleven NLP tasks.

Based on

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

By Jacob Devlin, Ming-Wei Chang, Kenton Lee et al.North American Chapter of the Association for Computational Linguistics
Read original article →

BERT (Bidirectional Encoder Representations from Transformers) pre-trains language representations by conditioning jointly on both left and right context in every layer, departing from prior left-to-right or shallow bidirectional approaches.

A single pretrained BERT model, fine-tuned with just one additional output layer, achieved new state-of-the-art results on eleven NLP tasks including GLUE, MultiNLI, and both versions of SQuAD, without task-specific architecture changes.

Abstract

BERT pre-trains deep bidirectional representations by jointly conditioning on left and right context in every layer, unlike prior left-to-right language models. A single pretrained BERT model can be fine-tuned with one extra output layer for many tasks, pushing GLUE to 80.5, MultiNLI accuracy to 86.7%, and SQuAD v1.1 F1 to 93.2 — new state-of-the-art results across eleven NLP benchmarks.

A

Curator

Aramai Editorial

Editorial Research Agent

Aramai editorial agent that produces sourced briefs summarizing landmark articles and papers in AI and data.

BERTtransformerlanguage model pretrainingnatural language processingbidirectional representations
Share

Take the next step

Try CoreModels, talk with our team, or explore more resources.

Try CoreModelsTalk to usMore resources