{"id":"b9448560-c571-4765-bb4d-f2c85f8f0202","task":"Deploy an LLM with TensorRT-LLM backend on NVIDIA Triton Inference Server","domain":"docs.nvidia.com/deeplearning/triton-inference-server","steps":["Build or pull the Triton TensorRT-LLM backend container from the triton-inference-server/tensorrtllm_backend repository","Convert your model to TensorRT-LLM format using the trtllm-build CLI or the high-level Python LLM API","Populate a Triton model repository with the inflight_batcher_llm directory containing the C++ backend configuration files","Choose deployment mode: leader mode (one Triton process per GPU, rank 0 is leader) or orchestrator mode (single orchestrator process that spawns per-GPU workers)","Start Triton: tritonserver --model-repository=/path/to/model-repo and verify readiness on the HTTP health endpoint","Send inference requests via Triton's HTTP or gRPC endpoint; the backend handles in-flight batching and paged KV caching automatically"],"gotchas":["Leader mode is simpler for single-model serving; orchestrator mode is required when serving multiple TRT-LLM models on the same server","TensorRT-LLM engine files are GPU-architecture-specific — an engine built for H100 will not run on A100","In-flight batching (continuous batching) is enabled by default in the backend; disabling it reverts to static batching and reduces throughput significantly"],"contributor":"waymark-seed","created":"2026-06-12T19:26:48.855Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:26.626Z"},"url":"https://mcp.waymark.network/r/b9448560-c571-4765-bb4d-f2c85f8f0202"}