prefix tuning
**Prefix Tuning and Prompt Tuning** are **parameter-efficient fine-tuning methods that prepend trainable continuous vectors (soft prompts) to the model's input or hidden states**, optimizing only these prefix parameters while keeping all model weights frozen — achieving task adaptation with as few as 0.01-0.1% trainable parameters.
**Prefix Tuning** (Li & Liang, 2021): Prepends trainable key-value pairs to every attention layer. For each layer l, trainable prefixes P_k^l ∈ R^(p×d) and P_v^l ∈ R^(p×d) are concatenated to the key and value matrices: K' = [P_k^l; K], V' = [P_v^l; V]. The model attends to these virtual prefix tokens as if they were part of the input, but their representations are directly optimized rather than derived from input embeddings. Prefix length p is typically 10-200 tokens.
**Prompt Tuning** (Lester et al., 2021): A simpler variant that prepends trainable embeddings only to the input layer (not every attention layer). Trainable soft prompt P ∈ R^(p×d) is concatenated to the input embeddings: X' = [P; X]. Only P is optimized. Simpler than prefix tuning but requires longer prefixes for equivalent performance.
**Comparison**:
| Method | Where | Trainable Params | Expressiveness |
|--------|-------|-----------------|---------------|
| **Prompt tuning** | Input embedding only | p × d | Lower |
| **Prefix tuning** | All attention layers K,V | 2 × L × p × d | Higher |
| **P-tuning v2** | All layers, optimized init | 2 × L × p × d | Highest |
| **LoRA** | Weight matrices (parallel) | 2 × r × d per matrix | High |
**Why Soft Prompts Work**: Soft prompts occupy a continuous optimization space unconstrained by the discrete vocabulary — they can represent "virtual tokens" that have no natural language equivalent but effectively steer model behavior. This continuous space is richer than hard prompt optimization (which is constrained to discrete token combinations) and allows gradient-based optimization.
**Reparameterization Trick**: Direct optimization of prefix parameters can be unstable (high-dimensional, poorly conditioned). Prefix tuning introduces a reparameterization: P = MLP(P') where P' is a smaller set of parameters and MLP is a two-layer feedforward network. After training, the MLP is discarded and only the final P values are kept. This stabilizes training by providing a smoother optimization landscape.
**Scaling Behavior**: Prompt tuning's effectiveness scales with model size. For T5-XXL (11B), prompt tuning matches full fine-tuning performance with only ~20K trainable parameters per task. For smaller models (<1B), the gap between prompt tuning and full fine-tuning is significant — soft prompts cannot compensate for limited model capacity.
**Multi-Task and Transfer**: Since prompts are small, multiple task-specific prompts can coexist with a single frozen model — enabling efficient multi-task serving. Prompts can also be composed: combining a style prompt with a task prompt, or transferring prompts across related tasks. Prompt interpolation (linear combination of two task prompts) can create intermediate task behaviors.
**Limitations**: Prompt tuning reduces effective context length by p tokens; performance is sensitive to initialization (random init works but pretrained-token init is better); and soft prompts are not interpretable — projecting them to nearest vocabulary tokens rarely produces meaningful text.
**Prefix tuning and prompt tuning pioneered the insight that task-specific knowledge can be encoded in a tiny set of continuous parameters that steer a frozen model's behavior — establishing the foundation for parameter-efficient fine-tuning and the separation of general capabilities from task-specific adaptation.**