low rank adaptation lora

**Low-Rank Adaptation (LoRA)** is **the parameter-efficient fine-tuning method that freezes pretrained model weights and trains low-rank decomposition matrices injected into each layer** — reducing trainable parameters by 100-1000× (from billions to millions) while matching or exceeding full fine-tuning quality, enabling fine-tuning of 70B models on single consumer GPU and rapid switching between task-specific adapters in production. **LoRA Mathematical Foundation:** - **Low-Rank Decomposition**: for weight matrix W ∈ R^(d×k), instead of updating W → W + ΔW, parameterize ΔW = BA where B ∈ R^(d×r), A ∈ R^(r×k), and rank r << min(d,k); reduces parameters from d×k to (d+k)×r - **Typical Ranks**: r=8-64 for most applications; r=8 sufficient for simple tasks, r=32-64 for complex reasoning; original model has effective rank 100-1000; low-rank assumption: task-specific adaptation lies in low-dimensional subspace - **Scaling Factor**: output scaled by α/r where α is hyperparameter (typically α=16-32); allows changing r without retuning learning rate; LoRA output: h = Wx + (α/r)BAx where x is input - **Initialization**: A initialized with random Gaussian (mean 0, small std), B initialized to zero; ensures ΔW=0 at start; model begins at pretrained state; gradual adaptation during training **Application to Transformer Layers:** - **Attention Matrices**: apply LoRA to Q, K, V, and output projection matrices; 4 LoRA modules per attention layer; most common configuration; captures task-specific attention patterns - **Feedforward Layers**: optionally apply to FFN up/down projections; doubles trainable parameters but improves quality on complex tasks; trade-off between efficiency and performance - **Layer Selection**: can apply to subset of layers (e.g., last 50%, or every other layer); reduces parameters further; minimal quality loss for many tasks; useful for extreme memory constraints - **Embedding Layers**: typically frozen; some methods (AdaLoRA) adapt embeddings for domain shift; increases parameters but handles vocabulary mismatch **Training Efficiency:** - **Parameter Reduction**: 70B model with LoRA r=16 on attention: 70B frozen + 40M trainable = 0.06% trainable; fits optimizer states in 2-4GB vs 280GB for full fine-tuning - **Memory Savings**: no need to store gradients for frozen weights; optimizer states only for LoRA parameters; enables fine-tuning 70B model on 24GB GPU (vs 8×80GB for full fine-tuning) - **Training Speed**: 20-30% faster than full fine-tuning due to fewer gradient computations; can use larger batch sizes with saved memory; wall-clock time often 2-3× faster - **Convergence**: typically requires same or fewer steps than full fine-tuning; learning rate 1e-4 to 5e-4 (higher than full fine-tuning); stable training with minimal hyperparameter tuning **Quality and Performance:** - **Benchmark Results**: matches full fine-tuning on GLUE, SuperGLUE within 0.5%; exceeds full fine-tuning on some tasks (less overfitting); RoBERTa-base with LoRA: 90.5 vs 90.2 GLUE score for full fine-tuning - **Instruction Tuning**: Llama 2 7B with LoRA on Alpaca dataset achieves 95% of full fine-tuning quality; 13B/70B models show even smaller gap; sufficient for most production applications - **Domain Adaptation**: particularly effective for domain shift (medical, legal, code); captures domain-specific patterns in low-rank subspace; often outperforms full fine-tuning by reducing overfitting - **Few-Shot Learning**: works well with small datasets (100-1000 examples); low parameter count acts as regularization; prevents overfitting that plagues full fine-tuning on small data **Deployment and Inference:** - **Adapter Switching**: store multiple LoRA adapters (40MB each for 7B model); load different adapter per request; enables multi-tenant serving with single base model; switch adapters in <100ms - **Adapter Merging**: can merge LoRA weights into base model: W' = W + BA; creates standalone model; no inference overhead; useful for single-task deployment - **Batched Inference**: serve multiple adapters in same batch using different LoRA weights per sequence; requires framework support (vLLM, TensorRT-LLM); maximizes GPU utilization in multi-tenant scenarios - **Inference Speed**: with merged weights, identical to base model; with separate adapters, 5-10% overhead from additional matrix multiplications; negligible for most applications **Advanced Variants and Extensions:** - **QLoRA**: combines LoRA with 4-bit quantization of base model; fine-tune 65B model on single 48GB GPU; maintains quality while reducing memory 4×; democratizes large model fine-tuning - **AdaLoRA**: adaptively allocates rank budget across layers and matrices; prunes low-importance singular values; achieves better quality at same parameter budget; requires more complex training - **LoRA+**: uses different learning rates for A and B matrices; improves convergence and final quality; simple modification with significant impact; lr_B = 16 × lr_A works well - **DoRA (Weight-Decomposed LoRA)**: decomposes weights into magnitude and direction; applies LoRA to direction only; narrows gap to full fine-tuning; slight memory increase **Production Best Practices:** - **Rank Selection**: start with r=16 for most tasks; increase to r=32-64 for complex reasoning or large distribution shift; diminishing returns beyond r=64; validate with small experiments - **Target Modules**: Q, K, V, O projections for attention-focused tasks; add FFN for knowledge-intensive tasks; embeddings only for vocabulary mismatch - **Learning Rate**: 1e-4 to 5e-4 typical range; higher than full fine-tuning (1e-5 to 1e-6); use warmup (3-5% of steps); cosine decay schedule - **Regularization**: LoRA acts as implicit regularization; additional dropout often unnecessary; weight decay 0.01-0.1 if overfitting observed Low-Rank Adaptation is **the technique that democratized large language model fine-tuning** — by reducing memory requirements by 100× while maintaining quality, LoRA enables researchers and practitioners to customize billion-parameter models on consumer hardware, fundamentally changing the economics and accessibility of LLM adaptation.