Efficient Memory Management for Large Language Model Serving with PagedAttention

The paper targets high-throughput serving of large language models, which requires batching many requests at once but is bottlenecked by the key-value (KV) cache memory each request needs, which is huge and grows and shrinks dynamically. Managed inefficiently, this memory is wasted through fragmentation and redundant duplication, which limits the achievable batch size. To address this, the authors propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques of operating systems, and build vLLM, an LLM serving system, on top of it.

vLLM achieves near-zero waste in KV cache memory and enables flexible sharing of the KV cache within and across requests to further reduce memory usage. In evaluations, it improves the throughput of popular LLMs by 2-4x at the same level of latency compared with state-of-the-art systems such as FasterTransformer and Orca, with the improvement more pronounced for longer sequences, larger models, and more complex decoding algorithms. This mattered for making LLM inference substantially more efficient and scalable, and the source code was released publicly.