Parameter-Efficient Fine-Tuning (PEFT) Methods Survey provides a comprehensive comparison of techniques that adapt large pretrained models to downstream tasks by modifying only a small fraction of parameters, covering the design space of where to add parameters, how many, and the tradeoffs between efficiency, quality, and flexibility.
PEFT Landscape:
| Family | Methods | Trainable % | Where Modified |
|---|---|---|---|
| Additive (serial) | Bottleneck adapters, AdapterFusion | 1-5% | After attention/FFN |
| Additive (parallel) | LoRA, AdaLoRA, DoRA | 0.1-1% | Parallel to weight matrices |
| Soft prompts | Prefix tuning, prompt tuning, P-tuning | 0.01-0.1% | Input/attention prefixes |
| Selective | BitFit (bias only), diff pruning | 0.05-1% | Subset of existing params |
| Reparameterization | LoRA, Compacter, KronA | 0.1-1% | Low-rank/structured updates |
Head-to-Head Comparison (on NLU benchmarks, similar parameter budgets):
| Method | GLUE Avg | Params | Inference Overhead | Composability |
|---|---|---|---|---|
| Full fine-tuning | 88.5 | 100% | None | N/A |
| LoRA (r=8) | 87.9 | 0.3% | Zero (merged) | Excellent |
| Prefix tuning (p=20) | 86.8 | 0.1% | Minor (extra tokens) | Good |
| Adapters | 87.5 | 1.5% | Some (extra layers) | Good |
| BitFit | 85.2 | 0.05% | Zero | N/A |
| Prompt tuning | 85.0 | 0.01% | Minor (extra tokens) | Excellent |
LoRA Dominance: LoRA has become the most widely used PEFT method due to: zero inference overhead (adapters merge into base weights), strong performance across tasks and model sizes, simple implementation, easy multi-adapter serving, and compatibility with quantization (QLoRA). Most recent PEFT innovation builds on LoRA.
LoRA Variants:
| Variant | Innovation | Benefit |
|---|---|---|
| QLoRA | 4-bit base model + BF16 adapters | Fine-tune 70B on single GPU |
| AdaLoRA | Adaptive rank per layer via SVD | Better parameter allocation |
| DoRA | Decompose into magnitude + direction | Closer to full fine-tuning |
| LoRA+ | Different learning rates for A and B | Faster convergence |
| rsLoRA | Rank-stabilized scaling | Better at high ranks |
| GaLore | Low-rank gradient projection | Reduce optimizer memory |
When PEFT Falls Short: Tasks requiring deep behavioral changes (safety alignment, fundamental capability acquisition), very small target datasets (overfitting risk with any method), and tasks where the base model lacks prerequisite knowledge (PEFT adapts existing capabilities, doesn't create new ones from scratch).
Multi-Task and Modular PEFT: Train separate adapters for different capabilities and compose them: adapter merging — average or weighted sum of multiple LoRA adapters; adapter stacking — apply adapters sequentially for layered capabilities; mixture of LoRAs — route inputs to different adapters based on task (similar to MoE but for adapters). This enables modular AI systems where capabilities are independently developed and composed.
Practical Recommendations: Start with LoRA (rank 8-16) as the default; increase rank for complex tasks or large domain shifts; use QLoRA when GPU memory is limited; consider full fine-tuning only when PEFT underperforms significantly and compute is available; always evaluate on held-out data from the target distribution.
The PEFT revolution has fundamentally changed the economics of LLM adaptation — transforming fine-tuning from a resource-intensive specialization requiring dedicated GPU clusters into an accessible operation performable on consumer hardware, democratizing the ability to customize foundation models for any application.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.