prompt tuning

Prompt tuning learns continuous "soft prompts" while keeping the base model frozen. **Mechanism**: Prepend learned embedding vectors to input, these vectors trained via backpropagation while model weights stay fixed, learned prompts encode task-specific information. **Comparison to fine-tuning**: No model weight changes (100% parameter efficient), store tiny vectors per task (KB vs GB), easily plug different tasks at inference, avoids catastrophic forgetting. **Architecture**: Soft prompt embeddings (typically 10-100 tokens) concatenated before input, trained end-to-end on task data, different prompts for different tasks share same base model. **Training**: Initialize from vocabulary embeddings or random, backpropagate through frozen model, task-specific losses. **Scaling properties**: Works better with larger models, smaller models may need more prompt length. **When to use**: Multi-task deployment with single model, limited compute for fine-tuning, need to preserve base model capabilities. **Comparison to LoRA**: LoRA modifies attention weights, prompt tuning only adds input, LoRA generally more capable but prompt tuning simpler. Both are complementary to full fine-tuning for efficient adaptation.