soft prompt optimization

**Soft prompt optimization** (also called **prompt tuning**) is a parameter-efficient fine-tuning technique that learns **continuous embedding vectors** (soft prompts) prepended to the model's input — optimizing these vectors through gradient descent to steer the frozen language model toward better task performance without modifying any of the model's own weights. **How Soft Prompts Work** - Instead of using natural language tokens as the prompt, soft prompts are **trainable continuous vectors** in the model's embedding space. - These vectors are initialized randomly or from text embeddings and then **optimized via backpropagation** on task-specific training data. - During inference, the soft prompt vectors are prepended to the input embeddings — the model processes them as if they were part of the input sequence. - The model's **own parameters remain frozen** — only the soft prompt vectors are updated. **Soft Prompt vs. Hard Prompt** - **Hard Prompt**: Discrete text tokens — human-readable, works with any API. - **Soft Prompt**: Continuous vectors — not human-readable, not constrained to correspond to any real words. Can represent concepts that have no direct textual equivalent. - Soft prompts are **more expressive** — they occupy a continuous space without the constraint of mapping to vocabulary tokens. **Soft Prompt Optimization Methods** - **Prompt Tuning (Lester et al.)**: Prepend $k$ learnable vectors (typically 20–100 tokens) to the input. Train on task data with cross-entropy loss. - **Prefix Tuning (Li & Liang)**: Prepend learnable vectors to the key and value matrices at every transformer layer — not just the input embedding. More parameters but greater influence on the model. - **P-Tuning**: Learn continuous prompts that can be inserted at arbitrary positions in the input, not just the beginning. - **P-Tuning v2**: Extends prefix tuning with per-layer learnable prompts — competitive with full fine-tuning on many tasks. **Benefits** - **Parameter Efficiency**: Only the soft prompt vectors are stored per task — typically **0.01–0.1%** of the model's parameters. One base model can serve many tasks with different small soft prompts. - **No Catastrophic Forgetting**: The model's weights are frozen — it retains all its general capabilities. Different tasks use different soft prompts with the same base model. - **Scalability**: As model size increases, prompt tuning performance approaches that of full fine-tuning — for large models (>10B parameters), the gap is very small. - **Storage Efficiency**: Each task requires only a few KB of prompt vectors, not a full model copy — enabling efficient multi-task deployment. **Challenges** - **Requires Model Access**: Need access to the model's embedding layer and gradients — doesn't work with black-box API-only models. - **Training Data Needed**: Requires labeled task data for optimization — not zero-shot. - **Initialization Sensitivity**: Performance depends on how the soft prompt vectors are initialized — text-based initialization often works better than random. - **Smaller Models**: For models under ~1B parameters, soft prompt tuning significantly underperforms full fine-tuning. Soft prompt optimization is a **key technique in efficient LLM adaptation** — it provides task specialization with minimal storage and compute overhead, enabling practical multi-task deployment of large language models.