retrieval-based demonstration
**Retrieval-Based Demonstration Selection** is the **technique of dynamically choosing few-shot examples from a candidate pool based on semantic similarity to the current input query — replacing random or static example selection with intelligent retrieval that provides the most relevant demonstrations for each inference instance** — the method that transforms few-shot prompting from a fragile, example-dependent process into a robust, input-adaptive system that consistently outperforms random selection by 5–20% across diverse NLP tasks.
**What Is Retrieval-Based Demonstration Selection?**
- **Definition**: At inference time, embedding the current input query and retrieving the top-k most semantically similar labeled examples from a pre-indexed demonstration pool to serve as few-shot context for the language model.
- **Core Insight**: The quality and relevance of few-shot examples matter enormously — examples similar to the current query activate more relevant in-context learning patterns than random examples.
- **Dynamic Per-Instance**: Unlike static few-shot prompts where every query sees the same examples, retrieval-based selection tailors the demonstrations to each specific input — adapting the prompt context on the fly.
- **Scalable Pool**: The demonstration pool can contain thousands of labeled examples without increasing prompt length — only the top-k retrieved examples appear in the prompt.
**Why Retrieval-Based Demonstration Selection Matters**
- **5–20% Accuracy Gain**: Research consistently shows significant improvements over random selection, especially on tasks with high input diversity (heterogeneous queries benefit most from tailored examples).
- **Robustness to Pool Size**: Performance improves as the demonstration pool grows — more candidates means higher probability of finding highly relevant examples for any query.
- **Reduces Sensitivity to Example Choice**: Random selection's high variance (different random examples → wildly different accuracy) is replaced by consistent, similarity-driven selection.
- **No Model Fine-Tuning**: Entirely prompt-level technique — works with any API-accessible LLM without gradient access or weight modification.
- **Production Ready**: Simple to implement with standard embedding models and vector databases — scales to thousands of QPS in production.
**Retrieval-Based Demonstration Pipeline**
**Offline Indexing**:
- Embed all candidate demonstrations (input + label) using a sentence embedding model (e.g., text-embedding-ada-002, all-MiniLM-L6-v2).
- Store embeddings in a vector index (FAISS, Pinecone, Chroma) for fast nearest-neighbor retrieval.
- Optionally cluster demonstrations to ensure pool diversity.
**Online Retrieval**:
- Embed the incoming query using the same embedding model.
- Retrieve top-k nearest neighbors from the demonstration pool by cosine similarity.
- Format retrieved demonstrations as few-shot examples in the prompt template.
**Prompt Assembly**:
- Order retrieved examples (nearest-last typically performs best — recency bias in attention).
- Prepend task instruction before the demonstrations.
- Append the current query after the demonstrations for the model to complete.
**Selection Strategies Comparison**
| Strategy | Selection Criterion | Performance | Latency Overhead |
|----------|-------------------|-------------|-----------------|
| **Random** | Uniform random from pool | Baseline (high variance) | None |
| **KATE (kNN)** | Embedding cosine similarity | +5–15% vs. random | ~5 ms (vector search) |
| **BM25** | Lexical overlap (TF-IDF) | +3–10% vs. random | ~2 ms |
| **Diverse kNN** | Similarity + MMR diversity | +7–20% vs. random | ~10 ms |
| **Learned Retriever** | Trained on downstream task | +10–25% vs. random | ~5 ms |
Retrieval-Based Demonstration Selection is **the production-grade solution to few-shot example quality** — replacing the lottery of random sampling with principled, similarity-driven retrieval that ensures every LLM inference benefits from the most relevant available demonstrations, making few-shot prompting reliable enough for enterprise deployment.