Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Driven by the high computational cost of running deep learning models on intelligent mobile devices, this paper proposes a quantization scheme that lets inference be performed with integer-only arithmetic, which can be implemented more efficiently than floating-point inference on widely available integer-only hardware. Crucially, the authors co-design a training procedure so that end-to-end model accuracy is preserved after the model is quantized.

The result is a better tradeoff between accuracy and on-device latency. The improvements are significant even on MobileNets—a model family already known for run-time efficiency—and the authors demonstrate the benefits on ImageNet classification and COCO object detection running on popular CPUs, making the approach practical for accurate on-device inference.