RAG, Retrieval, and Knowledge Bases

A comprehensive technical guide with mathematical foundations

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">RAG — Retrieval-Augmented Generation</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">ground an LLM in external knowledge: retrieve relevant docs, stuff into prompt, then generate</text>

  <!-- === MAIN PIPELINE (left to right) === -->
  <text x="380" y="74" fill="#e6edf3" font-size="12" font-weight="600" text-anchor="middle">RAG Pipeline (query → answer)</text>

  <!-- User query -->
  <rect x="30" y="90" width="95" height="40" rx="5" fill="#0f2a1f" stroke="#34d399" stroke-width="1.3"/>
  <text x="77" y="108" fill="#6ee7b7" font-size="9" text-anchor="middle" font-weight="600">User Query</text>
  <text x="77" y="122" fill="#8b98a5" font-size="7.5" text-anchor="middle">"How does EUV</text>
  <text x="77" y="131" fill="#8b98a5" font-size="7.5" text-anchor="middle">lithography work?"</text>

  <!-- Arrow to embed -->
  <path d="M127,110 L150,110" fill="none" stroke="#34d399" stroke-width="1.2"/>
  <polygon points="148,107 154,110 148,113" fill="#34d399"/>

  <!-- Embed query -->
  <rect x="156" y="92" width="72" height="36" rx="4" fill="#1c1633" stroke="#a78bfa" stroke-width="1.2"/>
  <text x="192" y="108" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">Embed</text>
  <text x="192" y="121" fill="#8b98a5" font-size="7.5" text-anchor="middle">q → vector</text>

  <!-- Arrow to search -->
  <path d="M230,110 L253,110" fill="none" stroke="#a78bfa" stroke-width="1.2"/>
  <polygon points="251,107 257,110 251,113" fill="#a78bfa"/>

  <!-- Vector DB search -->
  <rect x="259" y="84" width="105" height="52" rx="5" fill="#080d14" stroke="#60a5fa" stroke-width="1.4"/>
  <text x="311" y="102" fill="#93c5fd" font-size="9" text-anchor="middle" font-weight="600">Vector Search</text>
  <!-- Small vectors (dots in high-D space) -->
  <circle cx="278" cy="116" r="2.5" fill="#60a5fa" opacity="0.3"/>
  <circle cx="290" cy="120" r="2.5" fill="#60a5fa" opacity="0.4"/>
  <circle cx="300" cy="112" r="2.5" fill="#60a5fa" opacity="0.3"/>
  <circle cx="312" cy="118" r="3" fill="#fbbf24"/>
  <circle cx="322" cy="108" r="2.5" fill="#60a5fa" opacity="0.5"/>
  <circle cx="335" cy="115" r="2.5" fill="#60a5fa" opacity="0.3"/>
  <circle cx="345" cy="122" r="2.5" fill="#60a5fa" opacity="0.4"/>
  <text x="311" y="133" fill="#6b7684" font-size="7" text-anchor="middle">top-k nearest</text>

  <!-- Arrow to retrieved docs -->
  <path d="M366,110 L389,110" fill="none" stroke="#60a5fa" stroke-width="1.2"/>
  <polygon points="387,107 393,110 387,113" fill="#60a5fa"/>

  <!-- Retrieved chunks -->
  <rect x="395" y="84" width="100" height="52" rx="4" fill="#0f1a2a" stroke="#60a5fa" stroke-width="1.2"/>
  <text x="445" y="100" fill="#93c5fd" font-size="8.5" text-anchor="middle" font-weight="600">Retrieved Docs</text>
  <!-- Document snippets -->
  <rect x="403" y="105" width="82" height="8" rx="1" fill="#1e3a5f" opacity="0.6"/>
  <rect x="403" y="115" width="75" height="8" rx="1" fill="#1e3a5f" opacity="0.5"/>
  <rect x="403" y="125" width="80" height="8" rx="1" fill="#1e3a5f" opacity="0.4"/>
  <text x="445" y="143" fill="#6b7684" font-size="7" text-anchor="middle">k=3-10 chunks</text>

  <!-- Arrow to prompt assembly -->
  <path d="M497,110 L520,110" fill="none" stroke="#8b98a5" stroke-width="1.2"/>
  <polygon points="518,107 524,110 518,113" fill="#8b98a5"/>

  <!-- Augmented prompt -->
  <rect x="526" y="82" width="95" height="56" rx="5" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.3"/>
  <text x="573" y="98" fill="#fbbf24" font-size="8.5" text-anchor="middle" font-weight="600">Augmented</text>
  <text x="573" y="110" fill="#fbbf24" font-size="8.5" text-anchor="middle" font-weight="600">Prompt</text>
  <text x="573" y="124" fill="#8b98a5" font-size="7" text-anchor="middle">[system + docs</text>
  <text x="573" y="134" fill="#8b98a5" font-size="7" text-anchor="middle">+ user query]</text>

  <!-- Arrow to LLM -->
  <path d="M623,110 L646,110" fill="none" stroke="#f59e0b" stroke-width="1.2"/>
  <polygon points="644,107 650,110 644,113" fill="#f59e0b"/>

  <!-- LLM generates answer -->
  <rect x="652" y="88" width="78" height="44" rx="5" fill="#1c1633" stroke="#a78bfa" stroke-width="1.4"/>
  <text x="691" y="106" fill="#c4b5fd" font-size="10" text-anchor="middle" font-weight="600">LLM</text>
  <text x="691" y="120" fill="#8b98a5" font-size="7.5" text-anchor="middle">generate</text>
  <text x="691" y="130" fill="#6b7684" font-size="7" text-anchor="middle">grounded answer</text>

  <!-- === BOTTOM LEFT: Ingestion pipeline (offline) === -->
  <rect x="30" y="155" width="340" height="130" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="200" y="173" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Ingestion Pipeline (offline, build index)</text>

  <!-- Documents -->
  <rect x="45" y="185" width="55" height="35" rx="3" fill="#14202c" stroke="#3a4453" stroke-width="0.8"/>
  <text x="72" y="200" fill="#8b98a5" font-size="8" text-anchor="middle">Docs</text>
  <text x="72" y="212" fill="#6b7684" font-size="7" text-anchor="middle">PDF, web, DB</text>

  <!-- Arrow -->
  <path d="M102,202 L118,202" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="116,199 122,202 116,205" fill="#8b98a5"/>

  <!-- Chunk -->
  <rect x="124" y="185" width="55" height="35" rx="3" fill="#14202c" stroke="#3a4453" stroke-width="0.8"/>
  <text x="151" y="200" fill="#8b98a5" font-size="8" text-anchor="middle">Chunk</text>
  <text x="151" y="212" fill="#6b7684" font-size="7" text-anchor="middle">512 tokens</text>

  <!-- Arrow -->
  <path d="M181,202 L197,202" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="195,199 201,202 195,205" fill="#8b98a5"/>

  <!-- Embed -->
  <rect x="203" y="185" width="55" height="35" rx="3" fill="#1c1633" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="230" y="200" fill="#c4b5fd" font-size="8" text-anchor="middle">Embed</text>
  <text x="230" y="212" fill="#6b7684" font-size="7" text-anchor="middle">1536-dim</text>

  <!-- Arrow -->
  <path d="M260,202 L276,202" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="274,199 280,202 274,205" fill="#8b98a5"/>

  <!-- Vector DB store -->
  <rect x="282" y="185" width="75" height="35" rx="3" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.8"/>
  <text x="319" y="200" fill="#93c5fd" font-size="8" text-anchor="middle">Vector DB</text>
  <text x="319" y="212" fill="#6b7684" font-size="7" text-anchor="middle">Pinecone/Pgvector</text>

  <!-- Models used -->
  <text x="200" y="240" fill="#8b98a5" font-size="8.5" text-anchor="middle">Embedding models: OpenAI ada-002, Cohere embed, E5, BGE</text>
  <text x="200" y="255" fill="#6b7684" font-size="8" text-anchor="middle">Vector DBs: Pinecone, Weaviate, pgvector, Qdrant, Milvus</text>
  <text x="200" y="270" fill="#6b7684" font-size="8" text-anchor="middle">Chunking strategy: fixed-size, semantic, recursive</text>

  <!-- === BOTTOM RIGHT: Why RAG === -->
  <rect x="385" y="155" width="345" height="130" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="557" y="173" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Why RAG (vs fine-tuning)</text>

  <text x="405" y="195" fill="#34d399" font-size="9">✓ No retraining needed (plug in new docs)</text>
  <text x="405" y="211" fill="#34d399" font-size="9">✓ Grounded: cites sources, reduces hallucination</text>
  <text x="405" y="227" fill="#34d399" font-size="9">✓ Fresh knowledge (update index, not model)</text>
  <text x="405" y="243" fill="#34d399" font-size="9">✓ Access control: filter by user permissions</text>

  <text x="405" y="263" fill="#f87171" font-size="9">✗ Retrieval quality is the bottleneck</text>
  <text x="405" y="279" fill="#f87171" font-size="9">✗ Context window limits doc count</text>

  <!-- Advanced patterns -->
  <rect x="30" y="295" width="700" height="70" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="313" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Advanced RAG Patterns</text>
  <text x="50" y="333" fill="#c4b5fd" font-size="9">Hybrid search: dense + sparse (BM25)</text>
  <text x="280" y="333" fill="#60a5fa" font-size="9">Reranking: cross-encoder on top-k</text>
  <text x="510" y="333" fill="#34d399" font-size="9">Query rewriting: LLM expands query</text>
  <text x="50" y="351" fill="#fbbf24" font-size="9">Agentic RAG: multi-step retrieval</text>
  <text x="280" y="351" fill="#8b98a5" font-size="9">Self-RAG: model decides when to retrieve</text>
  <text x="510" y="351" fill="#8b98a5" font-size="9">GraphRAG: knowledge graph + vectors</text>

  <!-- Key numbers -->
  <text x="380" y="388" fill="#e6edf3" font-size="9.5" text-anchor="middle" font-weight="600">Production RAG: embed (1ms) → search (10ms) → rerank (50ms) → generate (500ms) — total ~600ms P50 latency</text>
  <text x="380" y="406" fill="#8b98a5" font-size="9" text-anchor="middle">Context stuffing: 3-10 chunks × 512 tokens = 1.5K-5K tokens of evidence injected into prompt</text>

  <!-- Scaling -->
  <text x="380" y="430" fill="#fbbf24" font-size="9.5" text-anchor="middle">ChatGPT uses RAG (web browsing), Perplexity is RAG-native, enterprise Q&amp;A = RAG over internal docs</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">RAG = give the LLM a library card. Instead of memorizing everything, it looks up what it needs at query time.</text>
</svg>

1. Overview

RAG (Retrieval-Augmented Generation) is an architecture that enhances Large Language Models (LLMs) by grounding their responses in external knowledge sources.

Core Components

• Generator: The LLM that produces the final response
• Retriever: The system that finds relevant documents
• Knowledge Base: The corpus of documents being searched

2. Mathematical Foundations

2.1 Vector Embeddings

Documents and queries are converted to dense vectors in $\mathbb{R}^d$ where $d$ is the embedding dimension (typically 384, 768, or 1536).

Embedding Function:

For a document $D$ and query $Q$:

2.2 Similarity Metrics

Cosine Similarity

Euclidean Distance (L2)

Dot Product

2.3 BM25 (Sparse Retrieval)

Where:

• $f(q_i, D)$ = frequency of term $q_i$ in document $D$
• $\lvert D \rvert$ = document length
• $\text{avgdl}$ = average document length in corpus
• $k_1$ = term frequency saturation parameter (typically 1.2–2.0)
• $b$ = length normalization parameter (typically 0.75)

Inverse Document Frequency (IDF):

Where:

• $N$ = total number of documents
• $n(q_i)$ = number of documents containing $q_i$

3. RAG Pipeline Architecture

3.1 Pipeline Stages

1. Indexing Phase

• Document ingestion
• Chunking strategy selection
• Embedding generation
• Vector storage

2. Query Phase

• Query embedding: $\vec{q} = E(Q)$
• Top-$k$ retrieval: $\mathcal{D}_k = \text{argmax}_{D \in \mathcal{C}}^k \text{sim}(\vec{q}, \vec{d})$
• Context assembly
• LLM generation

3.2 Retrieval Formula

Given a query $Q$ and corpus $\mathcal{C}$, retrieve top-$k$ documents:

3.3 Generation with Context

Where $\oplus$ denotes context concatenation.

4. Chunking Strategies

4.1 Fixed-Size Chunking

• Chunk size: $c$ tokens (typically 256–1024)
• Overlap: $o$ tokens (typically 10–20% of $c$)

4.2 Semantic Chunking

• Split by semantic boundaries (paragraphs, sections)
• Use sentence embeddings to detect topic shifts
• Threshold: $\theta$ for similarity drop detection

4.3 Recursive Chunking

• Hierarchical splitting: Document → Sections → Paragraphs → Sentences
• Maintains context hierarchy

5. Knowledge Base Design

5.1 Metadata Schema

{
  "chunk_id": "string",
  "document_id": "string",
  "content": "string",
  "embedding": "vector[d]",
  "metadata": {
    "source": "string",
    "title": "string",
    "author": "string",
    "date_created": "ISO8601",
    "date_modified": "ISO8601",
    "section": "string",
    "page_number": "integer",
    "chunk_index": "integer",
    "total_chunks": "integer",
    "tags": ["string"],
    "confidence_score": "float"
  }
}

5.2 Index Types

• Flat Index: Exact search, $O(n)$ complexity
• IVF (Inverted File): Approximate, $O(\sqrt{n})$ complexity
• HNSW (Hierarchical Navigable Small World): Graph-based, $O(\log n)$ complexity

HNSW Search Complexity:

Where $d$ is embedding dimension and $n$ is corpus size.

6. Evaluation Metrics

6.1 Retrieval Metrics

Recall@k

Precision@k

Mean Reciprocal Rank (MRR)

Normalized Discounted Cumulative Gain (NDCG)

6.2 Generation Metrics

• Faithfulness: Is response grounded in retrieved context?
• Relevance: Does response answer the query?
• Groundedness Score:

7. Advanced Techniques

7.1 Hybrid Search

Combine dense and sparse retrieval:

Where $\alpha \in [0, 1]$ is the weighting parameter.

7.2 Reranking

Apply cross-encoder reranking to top-$k$ results:

Cross-encoder complexity: $O(k \cdot \lvert Q \rvert \cdot \lvert D \rvert)$

7.3 Query Expansion

• HyDE (Hypothetical Document Embeddings):

• Multi-Query Retrieval:

7.4 Contextual Compression

Reduce retrieved context before generation:

8. Vector Database Options

9. Best Practices Checklist

• [ ] Choose appropriate chunk size based on content type
• [ ] Implement chunk overlap to preserve context
• [ ] Store rich metadata for filtering
• [ ] Use hybrid search for better recall
• [ ] Implement reranking for precision
• [ ] Monitor retrieval metrics continuously
• [ ] Evaluate groundedness of generated responses
• [ ] Handle edge cases (no results, low confidence)
• [ ] Implement caching for common queries
• [ ] Version control your knowledge base

10. Code Examples

10.1 Cosine Similarity (Python)

import numpy as np

def cosine_similarity(vec_q: np.ndarray, vec_d: np.ndarray) -> float:
    """
    Calculate cosine similarity between two vectors.

    $$\text{sim}_{\cos}(\vec{q}, \vec{d}) = \frac{\vec{q} \cdot \vec{d}}{\lVert \vec{q} \rVert \cdot \lVert \vec{d} \rVert}$$
    """
    dot_product = np.dot(vec_q, vec_d)
    norm_q = np.linalg.norm(vec_q)
    norm_d = np.linalg.norm(vec_d)
    return dot_product / (norm_q * norm_d)

10.2 BM25 Implementation

import math
from collections import Counter

def bm25_score(
    query_terms: list[str],
    document: list[str],
    corpus: list[list[str]],
    k1: float = 1.5,
    b: float = 0.75
) -> float:
    """
    Calculate BM25 score for a query-document pair.
    """
    doc_len = len(document)
    avg_doc_len = sum(len(d) for d in corpus) / len(corpus)
    doc_freq = Counter(document)
    N = len(corpus)

    score = 0.0
    for term in query_terms:
**Document frequency**
        n_q = sum(1 for d in corpus if term in d)

**IDF calculation**
        idf = math.log((N - n_q + 0.5) / (n_q + 0.5) + 1)

**Term frequency in document**
        f_q = doc_freq.get(term, 0)

**BM25 term score**
        numerator = f_q * (k1 + 1)
        denominator = f_q + k1 * (1 - b + b * (doc_len / avg_doc_len))

        score += idf * (numerator / denominator)

    return score

References

1. Lewis, P., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" 2. Robertson, S., & Zaragoza, H. (2009). "The Probabilistic Relevance Framework: BM25 and Beyond" 3. Johnson, J., et al. (2019). "Billion-scale similarity search with GPUs" (FAISS) 4. Malkov, Y., & Yashunin, D. (2018). "Efficient and robust approximate nearest neighbor search using HNSW"

Document generated for VS Code with KaTeX/LaTeX math support. Render with Markdown Preview Enhanced or similar extension.