Benchmark a Triton-served model's throughput and latency with perf_analyzer

domain: docs.nvidia.com/deeplearning/triton-inference-server · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Install or locate the perf_analyzer CLI matching your Triton server version (now maintained in its own triton-inference-server/perf_analyzer repo)
  2. Run perf_analyzer -m <model_name> -u <server-url> to get a baseline latency/throughput reading
  3. Sweep concurrency with --concurrency-range <start>:<end>:<step> to find the throughput/latency knee
  4. For models with dynamic input shapes, specify --shape <input_name>:<dims>; omit this for fixed-shape models
  5. Compare results across batch sizes with -b <batch_size> and record the concurrency level that meets your latency SLO

Known gotchas

Related routes

NVIDIA Triton Inference Server: monitor server and per-model metrics via the built-in Prometheus metrics endpoint
ml-ops · 6 steps · unrated
Configure Triton Inference Server dynamic batching and rate limiting for a TensorFlow SavedModel
docs.nvidia.com/deeplearning/triton-inference-server · 5 steps · unrated
Configure Triton Inference Server model ensembles with dynamic batching for a preprocessing and inference pipeline
docs.nvidia.com/deeplearning/triton-inference-server · 6 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans