demonstration selection

**Demonstration selection** is the process of choosing the **most effective in-context examples** (demonstrations) to include in a few-shot prompt — because the quality, relevance, and composition of the examples significantly impacts the language model's performance on the target task. **Why Demonstration Selection Matters** - In few-shot learning, the model learns the task pattern from the provided examples — **which examples are shown** can change accuracy by **10–20%** or more. - Random selection may include irrelevant, redundant, or misleading examples. - Strategic selection provides examples that are **maximally informative** for the specific input being processed. **Demonstration Selection Strategies** - **Similarity-Based Selection**: Choose examples most similar to the current test input. - **Embedding Similarity**: Compute sentence embeddings for all candidate examples and the test input. Select the $k$ nearest neighbors by cosine similarity. - **Intuition**: Similar examples demonstrate patterns most relevant to the current input — the model can more easily transfer the demonstrated pattern. - Most widely used and consistently effective approach. - **Diversity-Based Selection**: Choose examples that cover a wide range of the task space. - Select examples from different categories, different difficulty levels, different patterns. - Ensures the model sees the full scope of possible task behaviors. - Works well when the test input distribution is unknown. - **Similarity + Diversity**: Combine both — select examples that are relevant to the current input AND diverse among themselves. - **MMR (Maximal Marginal Relevance)**: Balance relevance to the query with diversity among selected examples. - **Difficulty-Based**: Choose examples with moderate difficulty. - Very easy examples may not be informative. Very hard or ambiguous examples may confuse the model. - Select examples where the model has moderate confidence — most informative for learning. - **Label-Balanced Selection**: Ensure the selected examples have a balanced distribution of labels/categories. - Imbalanced demonstrations can bias the model toward over-represented classes. **Advanced Selection Methods** - **Reinforcement Learning**: Train a selector model that chooses demonstrations to maximize downstream task performance. - **Influence Functions**: Estimate which training examples have the most positive influence on predicting the test input correctly. - **Iterative Selection**: Use the model's initial prediction to refine example selection — if the model is uncertain, select more relevant examples and retry. **Practical Considerations** - **Context Window**: Limited context length means typically 3–10 examples fit — selection quality matters more than quantity. - **Example Format**: Select examples that match the desired output format — the model imitates the demonstrated format. - **Recency**: Examples positioned later in the prompt (closer to the test input) may have more influence than earlier ones. Demonstration selection is one of the **highest-impact prompt engineering techniques** — systematic selection of few-shot examples can transform mediocre few-shot performance into state-of-the-art results.