Home Knowledge Base Mechanism

Prompt tuning learns continuous "soft prompts" while keeping the base model frozen. Mechanism: Prepend learned embedding vectors to input, these vectors trained via backpropagation while model weights stay fixed, learned prompts encode task-specific information. Comparison to fine-tuning: No model weight changes (100% parameter efficient), store tiny vectors per task (KB vs GB), easily plug different tasks at inference, avoids catastrophic forgetting. Architecture: Soft prompt embeddings (typically 10-100 tokens) concatenated before input, trained end-to-end on task data, different prompts for different tasks share same base model. Training: Initialize from vocabulary embeddings or random, backpropagate through frozen model, task-specific losses. Scaling properties: Works better with larger models, smaller models may need more prompt length. When to use: Multi-task deployment with single model, limited compute for fine-tuning, need to preserve base model capabilities. Comparison to LoRA: LoRA modifies attention weights, prompt tuning only adds input, LoRA generally more capable but prompt tuning simpler. Both are complementary to full fine-tuning for efficient adaptation.

prompt tuningfine-tuning

Explore 500+ Semiconductor & AI Topics

From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.