Triton Inference Server

What is Triton? NVIDIA Triton Inference Server is a production-grade serving platform that supports multiple frameworks, dynamic batching, and GPU orchestration.

Key Features

Model Repository Structure

<svg viewBox="0 0 452 207" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto" role="img"><rect x="0" y="0" width="452" height="207" rx="12" fill="#0d1117"/><g font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,&quot;Liberation Mono&quot;,monospace" font-size="14"><text xml:space="preserve" x="20" y="31.7"><tspan fill="#c9d1d9">model_repository/</tspan></text><text xml:space="preserve" x="20" y="50.7"><tspan fill="#6e7681">├──</tspan><tspan fill="#c9d1d9"> llama/</tspan></text><text xml:space="preserve" x="20" y="69.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">├──</tspan><tspan fill="#c9d1d9"> config.pbtxt        # Model configuration</tspan></text><text xml:space="preserve" x="20" y="88.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">└──</tspan><tspan fill="#c9d1d9"> 1/                   # Version 1</tspan></text><text xml:space="preserve" x="20" y="107.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">       </tspan><tspan fill="#6e7681">└──</tspan><tspan fill="#c9d1d9"> model.onnx       # Model file</tspan></text><text xml:space="preserve" x="20" y="126.7"><tspan fill="#6e7681">├──</tspan><tspan fill="#c9d1d9"> embeddings/</tspan></text><text xml:space="preserve" x="20" y="145.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">├──</tspan><tspan fill="#c9d1d9"> config.pbtxt</tspan></text><text xml:space="preserve" x="20" y="164.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">   </tspan><tspan fill="#6e7681">└──</tspan><tspan fill="#c9d1d9"> 1/</tspan></text><text xml:space="preserve" x="20" y="183.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">       </tspan><tspan fill="#6e7681">└──</tspan><tspan fill="#c9d1d9"> model.pt</tspan></text></g></svg>

Model Configuration

# config.pbtxt
name: "llama"
platform: "onnxruntime_onnx"
max_batch_size: 16

input [
  {
    name: "input_ids"
    data_type: TYPE_INT64
    dims: [ -1 ]  # Variable length
  }
]

output [
  {
    name: "logits"
    data_type: TYPE_FP32
    dims: [ -1, 32000 ]
  }
]

dynamic_batching {
  preferred_batch_size: [ 4, 8, 16 ]
  max_queue_delay_microseconds: 50000
}

Running Triton

# Start server
docker run --gpus all -p 8000:8000 -p 8001:8001
  -v /path/to/models:/models
  nvcr.io/nvidia/tritonserver:24.01-py3
  tritonserver --model-repository=/models

Client Usage

import tritonclient.http as httpclient

client = httpclient.InferenceServerClient("localhost:8000")

# Create input
inputs = [httpclient.InferInput("input_ids", [1, 10], "INT64")]
inputs[0].set_data_from_numpy(input_array)

# Infer
outputs = [httpclient.InferRequestedOutput("logits")]
response = client.infer("llama", inputs, outputs=outputs)
result = response.as_numpy("logits")

Dynamic Batching Triton automatically batches requests:

<svg viewBox="0 0 468 93" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto" role="img"><rect x="0" y="0" width="468" height="93" rx="12" fill="#0d1117"/><g font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,&quot;Liberation Mono&quot;,monospace" font-size="14"><text xml:space="preserve" x="20" y="31.7"><tspan fill="#c9d1d9">Request 1: batch_size=1  </tspan><tspan fill="#6e7681">─┐</tspan></text><text xml:space="preserve" x="20" y="50.7"><tspan fill="#c9d1d9">Request 2: batch_size=1  </tspan><tspan fill="#6e7681">─┼─►</tspan><tspan fill="#c9d1d9"> Combined batch_size=4</tspan></text><text xml:space="preserve" x="20" y="69.7"><tspan fill="#c9d1d9">Request 3: batch_size=2  </tspan><tspan fill="#6e7681">─┘</tspan></text></g></svg>

Benefits:

• Better GPU utilization
• Higher throughput
• Configurable latency trade-offs

Scaling

• Horizontal: Multiple Triton instances behind load balancer
• Multi-GPU: Multiple model instances across GPUs
• Kubernetes: Use Triton Inference Server Operator