Program Synthesis with Large Language Models

The paper explores the limits of large language models for general-purpose program synthesis by evaluating models ranging from 244M to 137B parameters on two new benchmarks. MBPP contains 974 entry-level Python tasks specified in natural language, while MathQA-Python contains 23,914 more complex problems. Models are tested in both few-shot and fine-tuning regimes, and the authors also study code dialog with human feedback and semantic grounding via execution prediction.

Synthesis performance scales log-linearly with model size: the largest model solves 59.6% of MBPP problems using few-shot prompting alone, fine-tuning adds roughly 10 percentage points across sizes, and the best fine-tuned model reaches 83.8% on MathQA-Python. Natural language feedback from a human halves the error rate relative to the initial prediction, yet even the strongest models generally cannot predict a program's output for a given input, exposing limits in their semantic grounding.