long context llm

**Long-Context LLMs** are the **large language model architectures and training techniques that extend the effective context window from the standard 2K-8K tokens to 128K, 1M, or beyond — enabling the model to process entire codebases, full-length books, hours of meeting transcripts, or massive document collections in a single forward pass**. **Why Context Length Is a Hard Problem** Standard transformer self-attention has O(n^2) time and memory complexity, where n is the sequence length. Doubling context length quadruples the attention computation. Additionally, positional encodings trained on short contexts often fail catastrophically at longer lengths, producing garbled outputs even if the compute budget is available. **Key Techniques** - **RoPE (Rotary Position Embedding) Scaling**: RoPE encodes positions as rotations in embedding space. By scaling the rotation frequencies — reducing them so the model "sees" longer sequences as slower rotations — a model trained on 4K tokens can generalize to 32K or 128K with minimal fine-tuning. YaRN and NTK-aware scaling refine the interpolation to preserve short-range attention precision. - **Ring Attention / Sequence Parallelism**: Distributes the long sequence across multiple GPUs, with each GPU computing attention only for its local chunk while ring-passing KV cache blocks to neighboring GPUs. This parallelizes the quadratic attention computation, enabling million-token contexts on multi-node clusters. - **Efficient Attention Variants**: FlashAttention computes exact attention without materializing the full n x n matrix, reducing memory from O(n^2) to O(n) while maintaining computational equivalence. Sliding window attention (Mistral) limits each token to attending only the nearest w tokens, trading global context for linear complexity. **The "Lost in the Middle" Problem** Even models with large context windows disproportionately attend to the beginning and end of the context, neglecting information placed in the middle. This is a training artifact: most training sequences are short, so the model has seen far more examples where the important information is near the edges. Explicit long-context fine-tuning with important facts randomly placed throughout the document is required to fix this retrieval pattern. **When to Use Long Context vs. RAG** - **Long Context**: Best when the full document must be understood holistically (summarization, complex reasoning across distant sections, code understanding). - **RAG**: Best when the relevant information is a small fraction of a massive corpus and the cost of encoding the entire corpus in one forward pass is prohibitive. Long-Context LLMs are **the architectural breakthrough that transforms language models from paragraph processors into document-scale reasoning engines** — unlocking applications that require understanding far beyond the traditional attention window.