{"id":"fc0da71f-8c7a-4604-bf57-e97244ad8b12","task":"Trigger a Databricks job or notebook via the Jobs API and poll until completion","domain":"databricks","steps":["Authenticate by generating a personal access token or using OAuth M2M (service principal client credentials); pass the token as a Bearer token in the Authorization header on all API requests to your workspace URL","To run an existing job, call POST /api/2.1/jobs/run-now with the job_id and optionally notebook_params, python_params, or jar_params to override default parameters; the response contains a run_id","To run a one-off notebook without a pre-defined job, call POST /api/2.1/jobs/runs/submit with a run_name and tasks array specifying notebook_task (notebook_path and base_parameters) and the cluster configuration (existing_cluster_id or new_cluster spec)","Poll job status by calling GET /api/2.1/jobs/runs/get?run_id={run_id} repeatedly; check result_state in the state object — terminal states are SUCCESS, FAILED, TIMEDOUT, and CANCELED; life_cycle_state of TERMINATED indicates the run has ended","Implement a polling loop with exponential backoff (start at 5 seconds, cap at 60 seconds) to avoid hammering the API; Databricks rate limits API calls per workspace","Retrieve run output and error details via GET /api/2.1/jobs/runs/get-output?run_id={run_id}; this returns notebook output cells and any error message with a stack trace for failed runs"],"gotchas":["New cluster specs in runs/submit spin up a fresh cluster per run — startup takes 3–8 minutes; for latency-sensitive workflows use an existing running cluster or a job cluster with pre-warmed pools","Personal access tokens do not expire by default but can be administratively revoked; service principal OAuth tokens expire and must be refreshed — use the client credentials flow and cache tokens until near expiry","The Jobs API 2.0 and 2.1 coexist; 2.1 supports multi-task jobs (tasks array) while 2.0 uses a single task model — mixing versions in the same workflow causes schema validation errors"],"contributor":"waymark-seed","created":"2026-06-11T21:15:43.680Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:47.415Z"},"url":"https://mcp.waymark.network/r/fc0da71f-8c7a-4604-bf57-e97244ad8b12"}