LoRA
LoRA (low-rank adaptation) is a fine-tuning method that freezes a model's original weights and trains small low-rank matrices added to its layers, so far fewer parameters change and much less GPU memory is needed.
Also called: low-rank adaptation, QLoRA
By Divya Sharma · Checked Nov 16, 2026 · How we review
Full fine-tuning updates every weight in a model, which needs memory for the weights, their gradients and the optimizer state. LoRA, introduced by Hu et al. in 2021, leaves the base weights untouched. For selected weight matrices it learns two thin matrices whose product is a low-rank update, and adds that update to the frozen weight. Only the thin matrices are trained.
Why builders use it
The trained part, called an adapter, is small. You can keep one base model and many adapters, one per customer or task, and swap them at load time. The paper reports that, compared with fully fine-tuning GPT-3 175B with Adam, LoRA cut trainable parameters by 10,000 times and GPU memory by 3 times. Because the update can be merged back into the weights, a merged LoRA model runs at the same speed as the original.
QLoRA
QLoRA (Dettmers et al., 2023) combines LoRA with quantization. The base model is stored frozen in 4-bit form and gradients flow through it into LoRA adapters kept in higher precision. The authors report fine-tuning a 65B parameter model on a single 48GB GPU this way. It is the usual route for tuning large open-weight models on one card.
Limits
LoRA restricts how much the model can change, so very large shifts in behaviour or a new language may still need full fine-tuning. The rank and which layers get adapters are settings you have to tune. LoRA is one of several parameter-efficient fine-tuning methods; tools such as Unsloth and Axolotl support it.
Related terms
- Fine-tuning
- Fine-tuning is further training of a pretrained model on a smaller, task-specific dataset, so the model adopts a particular skill, format, tone or domain without being trained from scratch.
- Parameter-efficient fine-tuning
- Parameter-efficient fine-tuning (PEFT) adapts a pretrained model by training only a small number of new or selected parameters, leaving the rest frozen. It cuts the memory, compute and storage that full fine-tuning needs.
- Quantization
- Quantization is a technique that stores a model's weights, and sometimes activations, with fewer bits, such as 8 or 4 instead of 16, so the model needs less memory and often runs faster, at some cost in accuracy.
- Supervised fine-tuning
- Supervised fine-tuning (SFT) trains a pretrained model further on example prompts paired with the responses you want. It is the usual first step that turns a base model into one that follows instructions.
- VRAM
- VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.