lora fine tuning

**Low-Rank Adaptation (LoRA)** is the **parameter-efficient fine-tuning technique that adds small, trainable low-rank decomposition matrices to frozen pretrained weights — factoring each weight update ΔW as the product of two small matrices (A and B) where ΔW = BA with rank r << d, reducing trainable parameters by 100-1000x while achieving fine-tuning quality comparable to full-parameter training**. **The Full Fine-Tuning Problem** Fine-tuning all parameters of a 70B model requires: 140 GB for weights (FP16), 140 GB for gradients, 280+ GB for optimizer states (Adam) = 560+ GB total memory. Each fine-tuned model is a separate 140 GB checkpoint. For organizations serving dozens of fine-tuned variants, the storage and memory costs are prohibitive. **How LoRA Works** For a pretrained weight matrix W ∈ R^(d×d): 1. **Freeze** W (no gradient computation or optimizer state needed) 2. **Add** a low-rank bypass: W' = W + ΔW = W + B·A, where B ∈ R^(d×r), A ∈ R^(r×d), and r << d (typically r = 8-64) 3. **Train** only A and B. For d=4096 and r=16: 2 × 4096 × 16 = 131K parameters per layer, vs. 4096² = 16.8M for the full weight. **128x reduction**. 4. **Scale**: ΔW is scaled by α/r to control the magnitude of the adaptation. **Which Layers to Adapt** Original LoRA applied adaptations to attention Q and V projection matrices only. Subsequent work showed that adapting all linear layers (Q, K, V, O projections + MLP up/down/gate projections) with appropriately small rank yields better results than adapting fewer layers with larger rank, for the same total parameter budget. **Practical Advantages** - **Memory Efficient**: Only A, B matrices and their optimizer states are stored in GPU memory. A LoRA fine-tune of Llama 70B with r=16 requires ~1 GB of trainable parameters (vs. 560 GB for full fine-tuning). - **Serving Efficiency**: Multiple LoRA adapters can share the same base model in production. Each request loads only the relevant LoRA weights (1-50 MB), switching between tasks in milliseconds. - **Merging**: After training, ΔW = BA can be computed and added permanently to W. The merged model is architecturally identical to the original — no inference overhead. This also enables model merging of multiple LoRAs. **Variants** - **QLoRA**: Combine LoRA with 4-bit quantization of the base model. The base weights are stored in NF4 (4-bit), while LoRA adapters are trained in BF16. Enables fine-tuning 65B models on a single 48GB GPU. - **DoRA (Weight-Decomposed Low-Rank Adaptation)**: Decomposes the weight update into magnitude and direction components, applying LoRA only to the direction. Consistently improves over standard LoRA, especially at low ranks. - **LoRA+**: Uses different learning rates for A and B matrices (B gets a higher rate), based on the observation that optimal learning dynamics differ for the two factors. LoRA is **the technique that made LLM fine-tuning accessible to everyone** — reducing the hardware requirement from a server rack to a single GPU by exploiting the empirical observation that the "change" needed to adapt a pretrained model to a new task lives in a remarkably low-dimensional subspace.