Large Language Models are Zero-Shot Reasoners

Pretrained large language models are generally treated as excellent few-shot learners, and chain-of-thought (CoT) prompting, which supplies step-by-step answer examples, had achieved state-of-the-art results on difficult multi-step reasoning tasks. This paper shows that LLMs are also decent zero-shot reasoners: by simply adding the phrase 'Let's think step by step' before each answer, a single prompt template called Zero-shot-CoT elicits multi-step reasoning without any hand-crafted few-shot exemplars.

Across diverse benchmarks spanning arithmetic (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical tasks, Zero-shot-CoT significantly outperforms standard zero-shot prompting, for example raising MultiArith accuracy from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with the large InstructGPT model, with similar gains on 540B-parameter PaLM. The versatility of one prompt across many tasks suggests broad, untapped zero-shot capabilities hidden inside LLMs and offers a strong, minimal baseline for reasoning benchmarks.