in-context retrieval

**In-context retrieval** is a technique in **Retrieval-Augmented Generation (RAG)** where relevant documents or knowledge are directly inserted into the model's **context window** (prompt), effectively using the LLM's input as a retrieval-augmented memory. Instead of fine-tuning the model on specific knowledge, you provide the information at inference time. **How It Works** - **Step 1 — Retrieve**: A retrieval system (vector search, keyword search, or hybrid) finds the most relevant documents or passages for the user's query. - **Step 2 — Inject**: The retrieved content is placed into the model's prompt, typically before the user's question, as context. - **Step 3 — Generate**: The LLM reads the injected context and generates a response that is **grounded** in the retrieved information. **Advantages** - **No Fine-Tuning Required**: Knowledge can be updated instantly by changing the retrieval corpus — no retraining needed. - **Reduced Hallucination**: The model can cite and reference specific retrieved passages rather than relying solely on parametric memory. - **Transparency**: Users can see exactly what documents the model used to form its answer. **Challenges** - **Context Window Limits**: Even with long-context models (128K+ tokens), there's a finite amount of information that can be injected. Retrieval quality is critical — irrelevant documents waste precious context space. - **Lost in the Middle**: Research shows LLMs pay more attention to information at the **beginning and end** of their context, sometimes missing relevant content in the middle. - **Retrieval Quality**: The system is only as good as the retriever — poor retrieval leads to poor or irrelevant responses. **Best Practices** - **Chunk Wisely**: Split documents into appropriately sized chunks that balance completeness with relevance. - **Rank and Filter**: Use a **reranker** to order retrieved chunks by relevance before context injection. - **Cite Sources**: Include metadata so the model can reference which document it drew information from.