Language Models are Few-Shot Learners

This paper investigates whether simply scaling up language models can substitute for task-specific fine-tuning. The authors train GPT-3, an autoregressive language model with 175 billion parameters, ten times larger than any previous non-sparse language model, and evaluate it in a few-shot setting where the model is given only a natural-language description of the task and a handful of demonstrations within its input text, with no gradient updates or fine-tuning applied at all.

GPT-3 achieves strong performance across a wide range of NLP tasks, including translation, question answering, and cloze-style completion, as well as tasks requiring on-the-fly reasoning or adaptation such as unscrambling words, using novel words in sentences, and simple arithmetic, at times becoming competitive with prior fine-tuned state-of-the-art methods. The paper also identifies datasets where GPT-3's few-shot learning still struggles, notes methodological issues from training on large web corpora, and finds that GPT-3 can produce news articles that human evaluators struggle to distinguish from human-written ones, prompting discussion of broader societal impacts.