Authenticate by generating a personal access token or using OAuth M2M (service principal client credentials); pass the token as a Bearer token in the Authorization header on all API requests to your workspace URL
To run an existing job, call POST /api/2.1/jobs/run-now with the job_id and optionally notebook_params, python_params, or jar_params to override default parameters; the response contains a run_id
To run a one-off notebook without a pre-defined job, call POST /api/2.1/jobs/runs/submit with a run_name and tasks array specifying notebook_task (notebook_path and base_parameters) and the cluster configuration (existing_cluster_id or new_cluster spec)
Poll job status by calling GET /api/2.1/jobs/runs/get?run_id={run_id} repeatedly; check result_state in the state object — terminal states are SUCCESS, FAILED, TIMEDOUT, and CANCELED; life_cycle_state of TERMINATED indicates the run has ended
Implement a polling loop with exponential backoff (start at 5 seconds, cap at 60 seconds) to avoid hammering the API; Databricks rate limits API calls per workspace
Retrieve run output and error details via GET /api/2.1/jobs/runs/get-output?run_id={run_id}; this returns notebook output cells and any error message with a stack trace for failed runs
Known gotchas
New cluster specs in runs/submit spin up a fresh cluster per run — startup takes 3–8 minutes; for latency-sensitive workflows use an existing running cluster or a job cluster with pre-warmed pools
Personal access tokens do not expire by default but can be administratively revoked; service principal OAuth tokens expire and must be refreshed — use the client credentials flow and cache tokens until near expiry
The Jobs API 2.0 and 2.1 coexist; 2.1 supports multi-task jobs (tasks array) while 2.0 uses a single task model — mixing versions in the same workflow causes schema validation errors
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp