Adapter-Based Fine-Tuning encompasses the family of parameter-efficient methods that add small trainable modules to a frozen pretrained model, enabling task-specific adaptation by updating only 0.1-5% of the total parameters — dramatically reducing memory, storage, and training cost compared to full fine-tuning while achieving competitive or equivalent task performance.

The Problem: Full fine-tuning of a 70B parameter model requires: ~280 GB for model weights (FP32), ~280 GB for gradients, ~560 GB for optimizer states (Adam) = ~1.1 TB of GPU memory. This is infeasible for most practitioners and creates separate model copies per task.

Adapter Taxonomy:

LoRA (Low-Rank Adaptation): The most widely adopted method. For a pretrained weight matrix W ∈ R^(d×k), LoRA adds a parallel low-rank update: W' = W + BA where B ∈ R^(d×r), A ∈ R^(r×k), and r << min(d,k) (typically r=4-64). During training, W is frozen and only A,B are updated. At inference, BA can be merged into W with zero overhead. The rank r controls the expressiveness-efficiency tradeoff.

LoRA Initialization and Training: A is initialized with random Gaussian, B with zeros (so the initial adaptation is zero — preserving pretrained behavior). Learning rate for LoRA is typically 5-10× higher than full fine-tuning learning rate. A scaling factor α/r is applied to the low-rank update to maintain consistent magnitude across different rank settings.

QLoRA: Combines quantization with LoRA — the base model is stored in 4-bit (NF4 quantization), LoRA adapters are trained in BF16, and gradients are computed through the quantized model using double quantization. This enables fine-tuning 65B models on a single 48GB GPU, democratizing LLM customization.

Multi-Adapter Serving: Because LoRA adapters are small (typically 10-100 MB vs. 100+ GB base model), multiple task-specific adapters can share a single base model in memory: load the base model once, swap LoRA weights per request. Systems like S-LoRA and Punica enable low-latency multi-adapter serving with batched inference across different adapters.

When to Use What: LoRA — general-purpose, best for most scenarios; full fine-tuning — when you have enough compute and need maximum quality; prefix tuning — when modifying attention patterns suffices; IA³ — when minimizing trainable parameters is paramount; adapter stacking — combine adapters trained on different capabilities.

Adapter-based fine-tuning has democratized LLM customization — enabling researchers and companies to specialize foundation models for their unique needs without the prohibitive cost of full fine-tuning, and establishing parameter efficiency as a fundamental design principle for the foundation model era.