LoRA: Low-Rank Adaptation of Large Language Models

This paper targets the growing infeasibility of full fine-tuning as pre-trained language models scale — deploying independent fine-tuned instances of a model like GPT-3 175B is prohibitively expensive. Low-Rank Adaptation (LoRA) freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of parameters trained for downstream tasks.

Compared to fine-tuning GPT-3 175B with Adam, LoRA reduces trainable parameters by 10,000 times and the GPU memory requirement by 3 times, while performing on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2, and GPT-3 — with higher training throughput and, unlike adapters, no additional inference latency. The paper also empirically investigates rank-deficiency in language model adaptation to explain LoRA's efficacy, and releases a package integrating LoRA with PyTorch along with implementations and model checkpoints.