Measuring Mathematical Problem Solving With the MATH Dataset

The paper introduces MATH, a benchmark of 12,500 challenging competition mathematics problems designed to measure mathematical problem-solving ability in machine learning models. Each problem comes with a full step-by-step solution, which can be used to train models to generate answer derivations and explanations. To help teach models the fundamentals of mathematics, the authors also contribute a large auxiliary pretraining dataset.

Even after improving accuracy, the authors find that performance on MATH remains relatively low, including for very large Transformer models. They observe that simply increasing compute budgets and parameter counts would be impractical for achieving strong mathematical reasoning if current scaling trends continue. Because scaling Transformers automatically solves most other text-based tasks but not MATH, they conclude that new algorithmic advances from the broader research community will likely be needed.