adapter layers

Adapter layers are small trainable modules inserted into frozen pretrained models, enabling parameter-efficient fine-tuning by learning task-specific transformations without modifying original weights. Architecture: typically bottleneck MLP with down-projection (reduce dimensions), nonlinearity, and up-projection (restore dimensions), added after transformer layers with residual connection. Parameters: typically 1-5% of base model—adapters might have 1M trainable params for 100M+ parameter base model. Training: freeze all pretrained parameters, only train adapter weights—drastically reduces compute and memory. Insertion points: after self-attention, after feed-forward, or both; add layer normalization before adapter. Bottleneck design: d → r → d where r << d (r often 64-256 for d=768-4096). Composition: multiple adapters for different tasks can be stacked or combined, enabling multi-task models. Comparison: full fine-tuning (all parameters—expensive, interference), adapter (small modules—efficient, modular), LoRA (low-rank weight updates—similar efficiency, different mechanism), and prefix tuning (learned prefix vectors). Multi-task: train separate adapters per task, share base model across all—efficient storage and inference. Adapter fusion: learn to combine multiple pretrained adapters for new tasks. Benefits: (1) single base model + multiple small adapters, (2) no catastrophic forgetting on pretrained knowledge, (3) efficient storage and deployment. Foundation for parameter-efficient transfer learning across NLP and vision.