LLM Inference Engines: vLLM and TGI

vLLM

What is vLLM? High-throughput LLM serving engine with PagedAttention for efficient KV cache management.

Key Features

Usage

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-7b-chat-hf")
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=256)

prompts = ["Hello, my name is", "The capital of France is"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.outputs[0].text)

API Server

# Start OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server
    --model meta-llama/Llama-2-7b-chat-hf
    --port 8000

# Use with OpenAI client
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
response = client.chat.completions.create(model="llama", messages=[...])

Text Generation Inference (TGI)

What is TGI? Hugging Face's production-ready LLM inference server, powering their Inference Endpoints.

Key Features

• Flash Attention 2 by default
• Continuous batching
• Quantization support (GPTQ, AWQ, bitsandbytes)
• Tensor parallelism for multi-GPU
• Built-in streaming

Running TGI

docker run --gpus all -p 8080:80
    -v /data:/data
    ghcr.io/huggingface/text-generation-inference:latest
    --model-id meta-llama/Llama-2-7b-chat-hf
    --quantize bitsandbytes-nf4

Client Usage

from huggingface_hub import InferenceClient

client = InferenceClient("http://localhost:8080")
response = client.text_generation(
    "What is deep learning?",
    max_new_tokens=100,
    stream=True
)
for token in response:
    print(token, end="", flush=True)

Comparison

When to Use

• vLLM: Max throughput, OpenAI-compatible API
• TGI: Hugging Face ecosystem, many quantization options