NVIDIA Triton Inference Server: High-Performance ML Serving

Model Repository Layout

Triton uses a directory-based model repository. Each model needs a config.pbtxt and versioned model files:

model_repository/
├── bert-onnx/
│   ├── config.pbtxt              # Model configuration
│   └── 1/
│       └── model.onnx            # Model file (version 1)
├── resnet50-tensorrt/
│   ├── config.pbtxt
│   └── 1/
│       └── model.plan            # TensorRT engine
└── gpt2-pytorch/
    ├── config.pbtxt
    └── 1/
        └── model.pt              # TorchScript model

# config.pbtxt for ONNX model
name: "bert-onnx"
backend: "onnxruntime"
max_batch_size: 64
input [
  {
    name: "input_ids"
    data_type: TYPE_INT64
    dims: [ -1 ]                  # -1 = dynamic sequence length
  },
  {
    name: "attention_mask"
    data_type: TYPE_INT64
    dims: [ -1 ]
  }
]
output [
  {
    name: "logits"
    data_type: TYPE_FP32
    dims: [ 2 ]
  }
]
dynamic_batching {
  preferred_batch_size: [ 16, 32, 64 ]
  max_queue_delay_microseconds: 5000
}
instance_group [
  { count: 2 kind: KIND_GPU gpus: [ 0 ] }
]

Dynamic Batching & Concurrent Execution

Start Triton and send inference requests via Python client:

# Start Triton (using NVIDIA Docker image)
# docker run --gpus all --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 \
#   -v $(pwd)/model_repository:/models \
#   nvcr.io/nvidia/tritonserver:24.01-py3 \
#   tritonserver --model-repository=/models

import tritonclient.http as httpclient
import numpy as np

# Create client
client = httpclient.InferenceServerClient(url="localhost:8000")
print("Server ready:", client.is_server_ready())
print("Model ready:", client.is_model_ready("bert-onnx"))

# Prepare inputs
input_ids = np.array([[101, 2023, 2003, 1037, 3231, 102]], dtype=np.int64)
attention_mask = np.ones_like(input_ids, dtype=np.int64)

inputs = [
    httpclient.InferInput("input_ids", input_ids.shape, "INT64"),
    httpclient.InferInput("attention_mask", attention_mask.shape, "INT64"),
]
inputs[0].set_data_from_numpy(input_ids)
inputs[1].set_data_from_numpy(attention_mask)

outputs = [httpclient.InferRequestedOutput("logits")]

# Send request — Triton dynamically batches with other concurrent requests
result = client.infer("bert-onnx", inputs, outputs=outputs)
logits = result.as_numpy("logits")
print(f"Predicted class: {logits.argmax()}")

Benchmarking with perf_analyzer

Find the optimal concurrency and batch size before production traffic hits:

# Install perf_analyzer (inside Triton container or via SDK)
# pip install tritonclient[all]

# Sweep concurrency from 1 to 16, measure for 5 seconds each
perf_analyzer \
  -m bert-onnx \
  -u localhost:8000 \
  --concurrency-range 1:16:2 \
  --measurement-interval 5000 \
  --input-data zero \
  --shape input_ids:1,128 \
  --shape attention_mask:1,128 \
  -f perf_results.csv

# Example output:
# Concurrency: 1, Throughput: 245 infer/sec, p95 latency: 12.3 ms
# Concurrency: 4, Throughput: 820 infer/sec, p95 latency: 14.1 ms
# Concurrency: 8, Throughput: 1240 infer/sec, p95 latency: 19.8 ms
# Concurrency: 16, Throughput: 1380 infer/sec, p95 latency: 35.6 ms
# → Sweet spot: concurrency=8 (1240 infer/sec, p95 < 20ms)

# Build TensorRT engine for 2-5x speedup
trtexec \
  --onnx=model.onnx \
  --saveEngine=model.plan \
  --fp16 \
  --minShapes=input_ids:1x64 \
  --optShapes=input_ids:16x128 \
  --maxShapes=input_ids:64x512

Triton vs TorchServe vs BentoML vs Ray Serve