QLoRA: Efficient Finetuning of Quantized LLMs

QLoRA is an efficient finetuning approach that reduces memory usage enough to finetune a 65-billion-parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. It works by backpropagating gradients through a frozen, 4-bit quantized pretrained language model into Low-Rank Adapters (LoRA). To save memory without sacrificing performance, it introduces three innovations: 4-bit NormalFloat (NF4), a data type that is information-theoretically optimal for normally distributed weights; double quantization, which quantizes the quantization constants to cut the average memory footprint; and paged optimizers to manage memory spikes.

Using QLoRA, the authors finetune more than 1,000 models, and their best family, named Guanaco, outperforms all previously released open models on the Vicuna benchmark, reaching 99.3% of ChatGPT's performance level after only 24 hours of finetuning on a single GPU. Their analysis shows that finetuning on a small, high-quality dataset can yield state-of-the-art results even with smaller models, that GPT-4 evaluations offer a cheap alternative to human evaluation, and that current chatbot benchmarks are not trustworthy. They release their models, code, and CUDA kernels for 4-bit training.