Adapter

An adapter is a small trainable module inserted into a pretrained neural network so that the network can learn a new task while its original weights stay frozen. The idea was presented for natural language processing by Neil Houlsby and colleagues at Google in the 2019 paper "Parameter-Efficient Transfer Learning for NLP", which tackled a practical headache: fine-tuning a BERT-sized model for every task produces a complete new copy per task, and a system serving dozens of tasks must store and manage dozens of full models. Adapters replace that with one shared base model plus a tiny task-specific add-on, making the added parameters per task a small fraction of the original model's size.

The adapter design in Houlsby's paper is a bottleneck: a down-projection to a small dimension, a nonlinearity, an up-projection back, and a residual connection around the whole thing. The narrow middle is what keeps the module cheap, since its size is governed by the bottleneck width rather than the full hidden dimension. Two adapters are placed in each transformer layer, one after the attention sub-layer and one after the feed-forward sub-layer, and they are initialized near the identity function so that the network starts out behaving like the original pretrained model. During training only the adapters, the layer normalization parameters and the task's output head are updated. Because the residual path passes the original signal through, the adapter only needs to learn a small correction.

The reported results made the case that most of the benefit of fine-tuning can be captured by a tiny fraction of the parameters. On the GLUE benchmark, Houlsby and colleagues found adapters came within about 0.4% of full fine-tuning performance while adding only about 3.6% parameters per task, and they also tested on a set of extra classification tasks. The approach also suits continual and multi-task setups: since the base weights never change, adding a new task cannot erase an old one, avoiding the catastrophic forgetting that sequential full fine-tuning invites. The adapter idea has roots in computer vision, where Rebuffi, Bilen and Vedaldi proposed residual adapters in 2017 for learning multiple visual domains with one network.

An ecosystem grew around adapters as composable parts. Pfeiffer and colleagues' AdapterFusion (2020) combined several task adapters with a learned attention mechanism to transfer knowledge between tasks without interference, and the AdapterHub framework from the same year offered a repository of pretrained adapters that could be downloaded and plugged in. Variants changed the placement, using a single adapter per layer, or added parallel rather than sequential adapter branches. These designs treat a model almost like a library with detachable skills, and the Hugging Face PEFT library now supports adapters alongside other methods.

Adapters differ from LoRA in an important practical way: inference cost. Because an adapter is a new set of layers in the forward path, it adds a small amount of sequential computation to every token, which is noticeable for latency-sensitive serving. LoRA, published in 2021 by Hu and colleagues, instead learns a low-rank update that can be folded back into the original weight matrix after training, so a merged LoRA model runs at exactly the speed of the base model. In exchange, adapters are modular by construction and can contain a nonlinearity, while LoRA's update is purely linear. Neither is strictly better, and both belong to the wider family of parameter-efficient fine-tuning.

A pitfall is to treat the word adapter as a single thing. It is used for bottleneck modules in the Houlsby sense, but practitioners also casually say adapter for any small trained add-on, including LoRA weights, so the file called an adapter in a model repository may be low-rank matrices rather than bottleneck layers. The distinction matters when merging, serving or comparing results: bottleneck adapters cannot simply be merged into the original weights, while LoRA adapters can.

Read an adapter as a removable plug-in skill fitted into a frozen model: a narrow bottleneck wired in beside the pretrained layers learns one task's adjustments, so the expensive base model is reused instead of copied.