lora
**LoRA (Low-Rank Adaptation)** is the **parameter-efficient fine-tuning technique that injects trainable low-rank decomposition matrices into frozen pretrained model weights** — enabling fine-tuning of large language models with 10,000× fewer trainable parameters than full fine-tuning, by approximating weight updates as a product of two small matrices (W = W₀ + BA where B ∈ R^(d×r), A ∈ R^(r×k), rank r ≪ min(d,k)), making it practical to adapt billion-parameter models on consumer GPUs.
**Core Idea: Low-Rank Weight Updates**
- Full fine-tuning: Update all W₀ ∈ R^(d×k) — too expensive for LLMs.
- LoRA insight: Weight updates during fine-tuning have low intrinsic rank — the update ΔW ≈ BA where r = 4–64 captures most useful adaptation.
- Merged at inference: W = W₀ + BA → no extra latency (matrices merged before deployment).
- Trainable params: r×(d+k) vs d×k. For d=k=4096, r=8: 65K vs 16M parameters.
**LoRA Architecture**
```python
import torch, torch.nn as nn
class LoRALinear(nn.Module):
def __init__(self, in_features, out_features, rank=8, alpha=16):
super().__init__()
self.W0 = nn.Linear(in_features, out_features, bias=False) # frozen
self.A = nn.Linear(in_features, rank, bias=False) # trainable
self.B = nn.Linear(rank, out_features, bias=False) # trainable
self.scale = alpha / rank # scaling factor
# Initialize: A ~ N(0,1), B = 0 (so LoRA starts at zero update)
nn.init.kaiming_uniform_(self.A.weight)
nn.init.zeros_(self.B.weight)
self.W0.weight.requires_grad = False # freeze base weights
def forward(self, x):
return self.W0(x) + self.scale * self.B(self.A(x))
```
**Where to Apply LoRA**
| Module | Typical in LLMs | Rank Recommendation |
|--------|----------------|--------------------|
| Q, V projection | Most common | r=8–32 |
| K projection | Sometimes | r=8–16 |
| FFN (MLP) layers | For stronger adaptation | r=16–64 |
| Embedding layer | For vocabulary expansion | r=4–8 |
**QLoRA: Quantized LoRA**
- QLoRA (Dettmers et al., 2023): Load pretrained model in 4-bit NF4 quantization → add LoRA adapters in bfloat16.
- NF4 (Normal Float 4-bit): Quantization levels chosen for normally distributed weights → minimal quantization error.
- Paged optimizers: Offload optimizer states to CPU RAM when GPU OOM → enables 65B model fine-tuning on single 48GB GPU.
- Typical result: QLoRA matches full 16-bit fine-tuning quality at ~30% GPU memory.
**Practical LoRA Settings**
```python
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=16, # rank
lora_alpha=32, # scaling (alpha/r = 2.0 is common)
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
lora_dropout=0.05, # regularization
bias="none", # don't train bias terms
task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters() # Shows << 1% trainable
```
**PEFT Method Comparison**
| Method | Params | Inference Overhead | Flexibility |
|--------|--------|--------------------|-------------|
| Full fine-tuning | 100% | 0% | Highest |
| LoRA | 0.1–2% | 0% (merged) | High |
| QLoRA | 0.1–2% | Low (4-bit base) | High |
| Prefix tuning | 0.1% | Small | Medium |
| Adapter layers | 1–5% | Small | Medium |
| IA3 | 0.01% | Minimal | Low |
**LoRA Variants**
- **DoRA (Weight-Decomposed LoRA)**: Decomposes weight into magnitude + direction; adapts direction via LoRA → better initialization.
- **LoRA+**: Different learning rates for A and B matrices → faster convergence.
- **AdaLoRA**: Adaptive rank allocation — important layers get higher rank, prunes unimportant singular values.
- **LoftQ**: Quantization-aware LoRA initialization — reduces gap between NF4 quantization and full precision.
LoRA and PEFT are **the enabling technology for democratizing large language model fine-tuning** — by reducing trainable parameters from billions to millions while preserving 95%+ of full fine-tuning quality, LoRA makes domain-specific LLM adaptation accessible on consumer hardware, turning what was a month-long distributed training job into an overnight single-GPU experiment and spawning the entire open-source fine-tuned LLM ecosystem.