Low-Rank Adaptation (LoRA) is the parameter-efficient fine-tuning technique that adds small, trainable low-rank decomposition matrices to frozen pretrained weights — factoring each weight update ΔW as the product of two small matrices (A and B) where ΔW = BA with rank r << d, reducing trainable parameters by 100-1000x while achieving fine-tuning quality comparable to full-parameter training.

The Full Fine-Tuning Problem

Fine-tuning all parameters of a 70B model requires: 140 GB for weights (FP16), 140 GB for gradients, 280+ GB for optimizer states (Adam) = 560+ GB total memory. Each fine-tuned model is a separate 140 GB checkpoint. For organizations serving dozens of fine-tuned variants, the storage and memory costs are prohibitive.

How LoRA Works

For a pretrained weight matrix W ∈ R^(d×d): 1. Freeze W (no gradient computation or optimizer state needed) 2. Add a low-rank bypass: W' = W + ΔW = W + B·A, where B ∈ R^(d×r), A ∈ R^(r×d), and r << d (typically r = 8-64) 3. Train only A and B. For d=4096 and r=16: 2 × 4096 × 16 = 131K parameters per layer, vs. 4096² = 16.8M for the full weight. 128x reduction. 4. Scale: ΔW is scaled by α/r to control the magnitude of the adaptation.

Which Layers to Adapt

Original LoRA applied adaptations to attention Q and V projection matrices only. Subsequent work showed that adapting all linear layers (Q, K, V, O projections + MLP up/down/gate projections) with appropriately small rank yields better results than adapting fewer layers with larger rank, for the same total parameter budget.

Practical Advantages

• Memory Efficient: Only A, B matrices and their optimizer states are stored in GPU memory. A LoRA fine-tune of Llama 70B with r=16 requires ~1 GB of trainable parameters (vs. 560 GB for full fine-tuning).
• Serving Efficiency: Multiple LoRA adapters can share the same base model in production. Each request loads only the relevant LoRA weights (1-50 MB), switching between tasks in milliseconds.
• Merging: After training, ΔW = BA can be computed and added permanently to W. The merged model is architecturally identical to the original — no inference overhead. This also enables model merging of multiple LoRAs.

Variants

• QLoRA: Combine LoRA with 4-bit quantization of the base model. The base weights are stored in NF4 (4-bit), while LoRA adapters are trained in BF16. Enables fine-tuning 65B models on a single 48GB GPU.
• DoRA (Weight-Decomposed Low-Rank Adaptation): Decomposes the weight update into magnitude and direction components, applying LoRA only to the direction. Consistently improves over standard LoRA, especially at low ranks.
• LoRA+: Uses different learning rates for A and B matrices (B gets a higher rate), based on the observation that optimal learning dynamics differ for the two factors.

LoRA is the technique that made LLM fine-tuning accessible to everyone — reducing the hardware requirement from a server rack to a single GPU by exploiting the empirical observation that the "change" needed to adapt a pretrained model to a new task lives in a remarkably low-dimensional subspace.