{"id":"0a330612-3e47-48e5-963f-618e7e65626d","task":"configure onnx runtime to use the tensorrt and cuda execution providers with fallback","domain":"onnxruntime.ai/docs/execution-providers","steps":["Append execution providers to SessionOptions in priority order, e.g. OrtSessionOptionsAppendExecutionProvider_Tensorrt then _CUDA, so ONNX Runtime falls back if TensorRT can't run a given op","Set TensorRT-specific options such as workspace size, and enable ORT_TENSORRT_FP16_ENABLE or ORT_TENSORRT_INT8_ENABLE if your model and hardware support reduced precision","Pass per-session execution provider options rather than relying only on environment variables, since explicit session options override environment-variable settings","Create the InferenceSession with the ordered provider list and confirm via session introspection which provider actually executed each node","Benchmark against CUDA-only and CPU-only execution to quantify TensorRT engine-build overhead on first run versus steady-state latency"],"gotchas":["TensorRT engine building happens lazily and can add significant first-inference latency — plan for engine cache/warm-up in latency-sensitive services","Provider order matters: ONNX Runtime falls back to the next provider in the list for unsupported ops, so the ordering changes which ops actually run on TensorRT","INT8 mode requires a calibration step; flipping ORT_TENSORRT_INT8_ENABLE without calibration data will not produce a valid accelerated engine"],"contributor":"waymark-seed","created":"2026-07-10T03:38:47.861Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/0a330612-3e47-48e5-963f-618e7e65626d"}