In-datacenter performance analysis of a tensor processing unit

Arguing that major cost-energy-performance gains must now come from domain-specific hardware, the authors evaluate the Tensor Processing Unit, a custom ASIC deployed in Google datacenters since 2015 to accelerate the inference phase of neural networks. The chip centers on a 65,536 8-bit multiply-accumulate matrix-multiply unit offering 92 TeraOps per second, backed by 28 MiB of software-managed on-chip memory. Its deterministic execution model is designed to meet strict 99th-percentile response-time requirements, unlike the throughput-oriented optimizations of CPUs and GPUs, which helps keep the TPU relatively small and low power.

Benchmarked against a server-class Intel Haswell CPU and an Nvidia K80 GPU from the same datacenters, using production TensorFlow workloads (MLPs, CNNs, and LSTMs that represent 95% of inference demand), the TPU ran on average about 15X to 30X faster with 30X to 80X higher performance per watt. The analysis also notes that adopting faster GDDR5 memory could triple achieved throughput. The study demonstrated the practical payoff of specialized accelerators and helped catalyze the industry's shift toward domain-specific hardware for deep learning.