prefix tuning

Prefix tuning is a parameter-efficient fine-tuning method that optimizes continuous task-specific vectors (prefixes) prepended to transformer layer inputs, adapting pretrained models without modifying original weights. Mechanism: instead of fine-tuning all model parameters, learn prefix embeddings P (matrix of learned vectors) prepended to keys and values in attention layers. Architecture: at each layer, prefix vectors are concatenated to the key/value sequences: Attention(Q, [P_k; K], [P_v; V]). Parameters: typically 0.1-1% of original model (e.g., 250K trainable for 350M model). Training: only prefix embeddings are trainable; all pretrained weights frozen. Comparison: (1) full fine-tuning (all parameters, expensive, requires storing per-task), (2) adapter layers (insert small MLPs, 3-4% params), (3) prefix tuning (only prefixes, u003c1%), (4) prompt tuning (simpler, only embeddings at input layer), (5) LoRA (low-rank adaptation of weight matrices). Advantages: minimal storage per task (one small matrix), preserves pretrained model completely, enables multi-task deployment. Performance: matches full fine-tuning on many tasks (summarization, table-to-text, translation). Implementation: typically parameterize prefix via small MLP for stable optimization. Extended: P-tuning v2 applies prefix tuning to all layers for better performance. Foundation for efficient LLM customization without full fine-tuning costs.