exllama
**ExLlama (and its successor ExLlamaV2)** is a **hyper-optimized Python/C++/CUDA inference engine specifically designed for maximum speed on NVIDIA GPUs** — writing custom CUDA kernels that bypass Hugging Face Transformers overhead to achieve the fastest possible inference for GPTQ and EXL2 quantized models, with ExLlamaV2 introducing the EXL2 format that enables mixed-precision quantization to perfectly fit any model into a specific VRAM budget.
**What Is ExLlama?**
- **Definition**: A CUDA-optimized inference library (created by turboderp) that implements LLM inference from scratch with custom GPU kernels — rather than using PyTorch's general-purpose operations, ExLlama writes specialized CUDA code for each operation in the transformer architecture, eliminating overhead.
- **Speed Leader**: Widely benchmarked as the fastest inference engine for quantized models on NVIDIA GPUs — achieving 2-3× higher tokens/second than Hugging Face Transformers with GPTQ models on the same hardware.
- **ExLlamaV2**: The complete rewrite that introduced the EXL2 quantization format — allowing mixed-precision quantization where different layers get different bit widths (e.g., attention layers at 5 bits, FFN layers at 3.5 bits) to optimally allocate a fixed VRAM budget.
- **EXL2 Format**: Unlike fixed-bitwidth quantization (all layers at 4-bit), EXL2 assigns bits per layer based on sensitivity — critical layers get more bits for quality, less important layers get fewer bits for compression. You specify a target bits-per-weight (e.g., 4.65 bpw) and the quantizer optimizes the allocation.
**Key Features**
- **Custom CUDA Kernels**: Hand-written CUDA kernels for quantized matrix multiplication, attention, RoPE, and layer normalization — each optimized for the specific memory access patterns of quantized inference.
- **Dynamic Batching**: ExLlamaV2 supports batched inference for serving multiple concurrent requests — essential for local API servers handling multiple users.
- **Speculative Decoding**: Use a small draft model to propose tokens verified by the main model — 2-3× speedup for generation with no quality loss.
- **Paged Attention**: Memory-efficient attention implementation that reduces VRAM waste from padding — enabling longer context lengths within the same VRAM budget.
- **Flash Attention Integration**: Uses Flash Attention 2 for the attention computation — combining ExLlama's quantized matmul kernels with Flash Attention's memory-efficient attention.
**ExLlamaV2 vs Other Inference Engines**
| Engine | Speed (NVIDIA) | Quantization | CPU Support | Ease of Use |
|--------|---------------|-------------|-------------|-------------|
| ExLlamaV2 | Fastest | GPTQ, EXL2 | No | Moderate |
| llama.cpp | Good | GGUF (all types) | Excellent | Easy |
| vLLM | Very fast | GPTQ, AWQ, FP16 | No | Easy (server) |
| Transformers | Baseline | GPTQ, AWQ, BnB | Yes | Easiest |
| TensorRT-LLM | Very fast | FP16, INT8, INT4 | No | Complex |
**ExLlama is the performance-maximizing inference engine for NVIDIA GPU users** — writing custom CUDA kernels that extract every possible token per second from quantized models, with ExLlamaV2's EXL2 format enabling precision-optimized quantization that perfectly fits any model into any VRAM budget.