adapter tuning
**Adapter-Based Fine-Tuning** encompasses the **family of parameter-efficient methods that add small trainable modules to a frozen pretrained model**, enabling task-specific adaptation by updating only 0.1-5% of the total parameters — dramatically reducing memory, storage, and training cost compared to full fine-tuning while achieving competitive or equivalent task performance.
**The Problem**: Full fine-tuning of a 70B parameter model requires: ~280 GB for model weights (FP32), ~280 GB for gradients, ~560 GB for optimizer states (Adam) = ~1.1 TB of GPU memory. This is infeasible for most practitioners and creates separate model copies per task.
**Adapter Taxonomy**:
| Method | Where Added | Trainable Params | Mechanism |
|--------|-----------|-----------------|----------|
| **LoRA** | Attention W_q, W_v (parallel) | 0.1-1% | Low-rank decomposition |
| **Bottleneck adapters** | After attention/FFN (serial) | 1-5% | Down-project → nonlinear → up-project |
| **Prefix tuning** | Prepend to K,V in attention | 0.1% | Virtual prefix tokens |
| **IA³** | Scale attention K,V and FFN | 0.01% | Learned rescaling vectors |
| **AdaLoRA** | Adaptive rank per layer | 0.1-1% | SVD-based rank allocation |
**LoRA (Low-Rank Adaptation)**: The most widely adopted method. For a pretrained weight matrix W ∈ R^(d×k), LoRA adds a parallel low-rank update: W' = W + BA where B ∈ R^(d×r), A ∈ R^(r×k), and r << min(d,k) (typically r=4-64). During training, W is frozen and only A,B are updated. At inference, BA can be merged into W with zero overhead. The rank r controls the expressiveness-efficiency tradeoff.
**LoRA Initialization and Training**: A is initialized with random Gaussian, B with zeros (so the initial adaptation is zero — preserving pretrained behavior). Learning rate for LoRA is typically 5-10× higher than full fine-tuning learning rate. A scaling factor α/r is applied to the low-rank update to maintain consistent magnitude across different rank settings.
**QLoRA**: Combines quantization with LoRA — the base model is stored in 4-bit (NF4 quantization), LoRA adapters are trained in BF16, and gradients are computed through the quantized model using double quantization. This enables fine-tuning 65B models on a single 48GB GPU, democratizing LLM customization.
**Multi-Adapter Serving**: Because LoRA adapters are small (typically 10-100 MB vs. 100+ GB base model), multiple task-specific adapters can share a single base model in memory: load the base model once, swap LoRA weights per request. Systems like S-LoRA and Punica enable low-latency multi-adapter serving with batched inference across different adapters.
**When to Use What**: **LoRA** — general-purpose, best for most scenarios; **full fine-tuning** — when you have enough compute and need maximum quality; **prefix tuning** — when modifying attention patterns suffices; **IA³** — when minimizing trainable parameters is paramount; **adapter stacking** — combine adapters trained on different capabilities.
**Adapter-based fine-tuning has democratized LLM customization — enabling researchers and companies to specialize foundation models for their unique needs without the prohibitive cost of full fine-tuning, and establishing parameter efficiency as a fundamental design principle for the foundation model era.**