PyTorch: An Imperative Style, High-Performance Deep Learning Library

This paper explains the design principles behind PyTorch, a machine learning library built to resolve the tradeoff between usability and speed seen in earlier deep learning frameworks. PyTorch supports an imperative and Pythonic programming style in which every part of the framework is a regular Python program under full user control, treating code itself as the model, which makes debugging straightforward and keeps it consistent with other scientific computing libraries.

The authors detail how PyTorch's runtime components are carefully and pragmatically implemented to work together, enabling efficient execution on hardware accelerators such as GPUs despite the flexible, dynamic programming model. They demonstrate the efficiency of individual subsystems as well as PyTorch's overall speed on several commonly used benchmarks, showing that usability and high performance need not be in conflict, which helped drive PyTorch's widespread adoption in research.