Evaluating Large Language Models Trained on Code

The paper introduces Codex, a GPT language model fine-tuned on publicly available code from GitHub, and studies its Python code-writing capabilities; a distinct production version of Codex powers GitHub Copilot. To measure functional correctness of programs synthesized from docstrings, the authors release HumanEval, a new evaluation set.

On HumanEval, Codex solves 28.8% of problems while GPT-3 solves 0% and GPT-J solves 11.4%, and repeated sampling proves surprisingly effective for difficult prompts, solving 70.2% of problems with 100 samples per problem. The paper also documents limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables, and discusses the broader impacts of deploying powerful code generation technologies across safety, security, and economics.