{"id":"f81430a1-356d-4f8c-9c58-9a08176a81c6","task":"ONNX Runtime: deploy a converted ONNX model behind a REST API (e.g. FastAPI) using an ONNX Runtime inference session","domain":"ml-ops","steps":["Install the runtime with `pip install onnxruntime` (CPU) or `pip install onnxruntime-gpu` (CUDA) — only one of the two packages should be installed in a given environment.","In your FastAPI app, create a single module-level `onnxruntime.InferenceSession(\"model.onnx\")` at startup (not per request) so the loaded graph is reused across requests.","Discover the expected tensor names once via `session.get_inputs()[0].name` and `session.get_outputs()[0].name` so you can build a correctly-keyed input dict.","Implement a POST endpoint that parses the request body into a numpy array of the right dtype/shape and calls `session.run(output_names_or_None, {input_name: input_array})`, returning the resulting array(s) as JSON.","Select an execution provider explicitly when it matters, e.g. `onnxruntime.InferenceSession(path, providers=[\"CPUExecutionProvider\"])`, since the provider determines which backend kernels run the graph.","Run the app with uvicorn and validate it via curl or the framework's interactive API docs before containerizing for deployment."],"gotchas":["Creating a new InferenceSession per request is expensive (model/graph load overhead) — construct it once and share it across requests/workers.","Input arrays must match the ONNX graph's exact dtype (commonly float32) and shape; passing e.g. float64 numpy arrays or mismatched shapes raises a runtime error from session.run.","Only one of onnxruntime (CPU) or onnxruntime-gpu should be installed in the same environment — installing both can cause import/execution-provider conflicts."],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/f81430a1-356d-4f8c-9c58-9a08176a81c6"}