vllm
**LLM Inference Engines: vLLM and TGI**
**vLLM**
**What is vLLM?**
High-throughput LLM serving engine with PagedAttention for efficient KV cache management.
**Key Features**
| Feature | Description |
|---------|-------------|
| PagedAttention | Non-contiguous KV cache, like virtual memory |
| Continuous batching | Add/remove requests dynamically |
| High throughput | 24x higher than HuggingFace baseline |
| OpenAI-compatible API | Drop-in replacement |
**Usage**
```python
from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-2-7b-chat-hf")
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=256)
prompts = ["Hello, my name is", "The capital of France is"]
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
print(output.outputs[0].text)
```
**API Server**
```bash
# Start OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server
--model meta-llama/Llama-2-7b-chat-hf
--port 8000
```
```python
# Use with OpenAI client
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
response = client.chat.completions.create(model="llama", messages=[...])
```
**Text Generation Inference (TGI)**
**What is TGI?**
Hugging Face's production-ready LLM inference server, powering their Inference Endpoints.
**Key Features**
- Flash Attention 2 by default
- Continuous batching
- Quantization support (GPTQ, AWQ, bitsandbytes)
- Tensor parallelism for multi-GPU
- Built-in streaming
**Running TGI**
```bash
docker run --gpus all -p 8080:80
-v /data:/data
ghcr.io/huggingface/text-generation-inference:latest
--model-id meta-llama/Llama-2-7b-chat-hf
--quantize bitsandbytes-nf4
```
**Client Usage**
```python
from huggingface_hub import InferenceClient
client = InferenceClient("http://localhost:8080")
response = client.text_generation(
"What is deep learning?",
max_new_tokens=100,
stream=True
)
for token in response:
print(token, end="", flush=True)
```
**Comparison**
| Feature | vLLM | TGI |
|---------|------|-----|
| PagedAttention | ✅ Native | ✅ Supported |
| OpenAI API | ✅ Built-in | ❌ Different API |
| Quantization | Limited | ✅ Extensive |
| Multi-GPU | ✅ Tensor parallel | ✅ Tensor parallel |
| Speculative decoding | ✅ | ✅ |
| Ease of use | Very easy | Easy |
**When to Use**
- **vLLM**: Max throughput, OpenAI-compatible API
- **TGI**: Hugging Face ecosystem, many quantization options