tgi

TGI (Text Generation Inference) is Hugging Face's production-grade inference server for large language models, implementing continuous batching, quantization, tensor parallelism, and other optimizations for high-throughput, low-latency serving. Core features: continuous batching (add/remove requests mid-batch), PagedAttention-style KV cache management, and dynamic batching for optimal GPU utilization. Quantization support: bitsandbytes (INT8/INT4), GPTQ, AWQ, and EETQ for reduced memory and faster inference; models can be loaded quantized directly. Tensor parallelism: split large models across multiple GPUs; serves models larger than single GPU memory. Hardware support: NVIDIA GPUs, AMD GPUs (ROCm), and Intel Gaudi accelerators. Deployment: Docker container, Kubernetes ready, and Inference Endpoints integration on Hugging Face Hub. API: OpenAI-compatible API endpoints for easy migration; streaming and non-streaming responses. Model support: most Hugging Face Transformers models; optimized paths for popular architectures (Llama, Mistral, Falcon, etc.). Speculative decoding: optional draft model for faster generation. Monitoring: Prometheus metrics for latency, throughput, and queue depth. Comparison: faster than naive Transformers, competitive with vLLM and TensorRT-LLM; strong open-source option with Hugging Face ecosystem integration. TGI provides production-ready open-source LLM serving.