Long Context LLM Techniques is methods extending large language model context length beyond original training window, enabling processing of longer documents while maintaining computational efficiency — essential for document understanding, code analysis, and long-form generation. Long context directly enables practical applications. Rotary Position Embeddings (RoPE) encodes position as rotation in complex plane rather than absolute position. Naturally extrapolates to longer sequences than training length. Position i is represented as rotation by angle θ_j i where θ_j = 10000^(-2j/d) with j varying over dimensions. Relative position information preserved through rotation differences. No learnable position parameters—purely geometric encoding. ALiBi (Attention with Linear Biases) adds linear bias to attention scores based on distance: bias = -α |i - j| where α is learnable per attention head. Simpler than positional embeddings, highly extrapolatable to longer sequences. Works across popular transformer architectures. No additional parameters compared to absolute position embeddings. Streaming LLM (Efficient Attention) maintains fixed-length attention window: only attend to recent K tokens plus few cached tokens. Compresses older attention values into summary cache (e.g., mean or attention-weighted summary), enabling constant memory growth with sequence length. Sparse Attention Patterns reduce quadratic attention complexity. Local attention: only attend to neighboring tokens (window). Strided attention: attend to every kth token. Combined patterns enable attending to global and local context. Linformer reduces attention from O(n²) to O(n). KV Cache Compression stores (key, value) pairs for all previously generated tokens to speed inference, but cache grows with sequence length. Quantization reduces cache size. Multi-query attention shares key/value across query heads. Group query attention shares across group of query heads. Hierarchical Processing processes document in chunks, summarizes chunks, attends to chunk summaries then details. Reduces attention span needed. Retrieval Augmentation instead of extending context, retrieve relevant chunks from external database. Transforms long-context problem to retrieval ranking. Popular in hybrid retrieval-generation systems. Training Techniques continued pretraining on longer sequences fine-tunes position embeddings, gradient checkpointing reduces memory, flash attention speeds computation. Inference Optimization batching multiple sequences, paging (memory manager for KV cache), speculative decoding (verify candidate tokens). Evaluation and Benchmarks needle-in-haystack tasks test long-context understanding, long-document QA datasets. Long context LLMs enable processing documents, code, books without splitting critical for practical applications requiring global understanding.