In-context retrieval is a technique in Retrieval-Augmented Generation (RAG) where relevant documents or knowledge are directly inserted into the model's context window (prompt), effectively using the LLM's input as a retrieval-augmented memory. Instead of fine-tuning the model on specific knowledge, you provide the information at inference time.

How It Works

• Step 1 — Retrieve: A retrieval system (vector search, keyword search, or hybrid) finds the most relevant documents or passages for the user's query.
• Step 2 — Inject: The retrieved content is placed into the model's prompt, typically before the user's question, as context.
• Step 3 — Generate: The LLM reads the injected context and generates a response that is grounded in the retrieved information.

Advantages

• No Fine-Tuning Required: Knowledge can be updated instantly by changing the retrieval corpus — no retraining needed.
• Reduced Hallucination: The model can cite and reference specific retrieved passages rather than relying solely on parametric memory.
• Transparency: Users can see exactly what documents the model used to form its answer.

Challenges

• Context Window Limits: Even with long-context models (128K+ tokens), there's a finite amount of information that can be injected. Retrieval quality is critical — irrelevant documents waste precious context space.
• Lost in the Middle: Research shows LLMs pay more attention to information at the beginning and end of their context, sometimes missing relevant content in the middle.
• Retrieval Quality: The system is only as good as the retriever — poor retrieval leads to poor or irrelevant responses.

Best Practices

• Chunk Wisely: Split documents into appropriately sized chunks that balance completeness with relevance.
• Rank and Filter: Use a reranker to order retrieved chunks by relevance before context injection.
• Cite Sources: Include metadata so the model can reference which document it drew information from.