long

**Long Context LLM Techniques** is **methods extending large language model context length beyond original training window, enabling processing of longer documents while maintaining computational efficiency** — essential for document understanding, code analysis, and long-form generation. Long context directly enables practical applications. **Rotary Position Embeddings (RoPE)** encodes position as rotation in complex plane rather than absolute position. Naturally extrapolates to longer sequences than training length. Position i is represented as rotation by angle θ_j * i where θ_j = 10000^(-2j/d) with j varying over dimensions. Relative position information preserved through rotation differences. No learnable position parameters—purely geometric encoding. **ALiBi (Attention with Linear Biases)** adds linear bias to attention scores based on distance: bias = -α * |i - j| where α is learnable per attention head. Simpler than positional embeddings, highly extrapolatable to longer sequences. Works across popular transformer architectures. No additional parameters compared to absolute position embeddings. **Streaming LLM (Efficient Attention)** maintains fixed-length attention window: only attend to recent K tokens plus few cached tokens. Compresses older attention values into summary cache (e.g., mean or attention-weighted summary), enabling constant memory growth with sequence length. **Sparse Attention Patterns** reduce quadratic attention complexity. Local attention: only attend to neighboring tokens (window). Strided attention: attend to every kth token. Combined patterns enable attending to global and local context. Linformer reduces attention from O(n²) to O(n). **KV Cache Compression** stores (key, value) pairs for all previously generated tokens to speed inference, but cache grows with sequence length. Quantization reduces cache size. Multi-query attention shares key/value across query heads. Group query attention shares across group of query heads. **Hierarchical Processing** processes document in chunks, summarizes chunks, attends to chunk summaries then details. Reduces attention span needed. **Retrieval Augmentation** instead of extending context, retrieve relevant chunks from external database. Transforms long-context problem to retrieval ranking. Popular in hybrid retrieval-generation systems. **Training Techniques** continued pretraining on longer sequences fine-tunes position embeddings, gradient checkpointing reduces memory, flash attention speeds computation. **Inference Optimization** batching multiple sequences, paging (memory manager for KV cache), speculative decoding (verify candidate tokens). **Evaluation and Benchmarks** needle-in-haystack tasks test long-context understanding, long-document QA datasets. **Long context LLMs enable processing documents, code, books without splitting** critical for practical applications requiring global understanding.