Bleu: a Method for Automatic Evaluation of Machine Translation

The paper tackles the problem that human evaluation of machine translation quality, while thorough, is expensive, slow (taking months), and produces labor that cannot be reused for future evaluations. To address this, the authors propose a method for automatic evaluation of machine translation that is quick to run, inexpensive, and independent of the target language, so it can be applied broadly across translation systems and languages.

The central result is that this automatic method correlates highly with human evaluation while having little marginal cost per additional run, making it usable as an automated understudy that substitutes for skilled human judges whenever quick or frequent evaluations are needed. This mattered because it gave the machine translation community a practical, reusable, and scalable way to measure progress and compare systems without requiring costly human evaluation each time, a metric (BLEU) that became a standard in the field.