Parameter-Efficient Fine-Tuning (PEFT) is the family of techniques that adapts large pre-trained models to downstream tasks by modifying only a small fraction (0.01-5%) of total parameters — achieving comparable performance to full fine-tuning while reducing memory requirements, training time, and storage costs by orders of magnitude.

LoRA (Low-Rank Adaptation):

• Mechanism: freezes the pre-trained weight matrix W (d×d) and adds a low-rank decomposition: ΔW = B·A where A is d×r and B is r×d with rank r ≪ d (typically r=4-64); the forward pass computes (W + ΔW)·x using only r×2×d trainable parameters instead of d² full parameters
• Weight Merging: at inference, ΔW = B·A is computed once and merged with W, producing zero additional inference latency; the adapted model has identical architecture and speed as the original — no architectural modifications needed at serving time
• Target Modules: typically applied to attention projection matrices (Q, K, V, O) and optionally MLP layers; applying LoRA to all linear layers (QLoRA-style) with very low rank (r=4) provides broad adaptation with minimal parameters
• QLoRA: combines LoRA with 4-bit NormalFloat quantization of the frozen base model; enables fine-tuning 65B parameter models on a single 48GB GPU; the base model is quantized (NF4) while LoRA adapters are trained in BF16

Other PEFT Methods:

• Adapter Layers: small bottleneck MLP modules inserted between Transformer layers; each adapter has down-projection (d→r), nonlinearity, and up-projection (r→d); adds ~2% parameters and slight inference latency from additional computation
• Prefix Tuning: prepends learnable continuous vectors (soft prompts) to the key/value sequences in each attention layer; the model's behavior is steered by these learned prefix embeddings rather than modifying weights; analogous to giving the model a task-specific instruction in its internal representation
• Prompt Tuning: simpler variant that only prepends learnable tokens to the input embedding layer (not every attention layer); fewer parameters than prefix tuning but less expressive; becomes competitive with full fine-tuning as model size increases beyond 10B parameters
• IA³ (Few-Parameter Fine-Tuning): learns three rescaling vectors that element-wise multiply keys, values, and FFN intermediate activations; only 3×d parameters per layer — among the most parameter-efficient methods with competitive performance

Practical Advantages:

• Multi-Task Serving: one base model serves multiple tasks by swapping lightweight adapters (2-50 MB each vs 14-140 GB for full model copies); adapter hot-swapping enables serving thousands of personalized models from a single GPU
• Memory Efficiency: full fine-tuning of Llama-70B requires ~140GB for model + ~420GB for optimizer states + gradients (BF16+FP32); QLoRA reduces this to ~35GB (4-bit model) + ~2GB (LoRA gradients) = single-GPU feasible
• Catastrophic Forgetting: PEFT methods partially mitigate catastrophic forgetting because the pre-trained weights are frozen; the model retains base capabilities while adapting to the target task through the small adapter parameters
• Training Stability: fewer trainable parameters produce smoother loss landscapes; PEFT training is typically more stable than full fine-tuning, requiring less hyperparameter tuning and fewer training iterations

Comparison:

• LoRA vs Full Fine-Tuning: LoRA achieves 95-100% of full fine-tuning performance for most tasks at r=16-64; gap is larger for tasks requiring significant knowledge update (domain-specific, multilingual); larger rank r closes the gap at the cost of more parameters
• LoRA vs Adapter: LoRA has zero inference overhead (merged weights); adapters add ~5-10% inference latency from additional forward passes; LoRA is preferred for serving efficiency
• LoRA vs Prompt Tuning: LoRA is more expressive and consistently outperforms prompt tuning for smaller models (<10B); prompt tuning approaches LoRA performance at very large scale and is simpler to implement

PEFT methods, especially LoRA, have democratized large model fine-tuning — enabling individual researchers and small teams to customize state-of-the-art models on consumer hardware, making the personalization and specialization of billion-parameter models accessible to the entire AI community.