Trigger a Databricks job or notebook via the Jobs API and poll until completion

Steps

1. Authenticate by generating a personal access token or using OAuth M2M (service principal client credentials); pass the token as a Bearer token in the Authorization header on all API requests to your workspace URL
2. To run an existing job, call POST /api/2.1/jobs/run-now with the job_id and optionally notebook_params, python_params, or jar_params to override default parameters; the response contains a run_id
3. To run a one-off notebook without a pre-defined job, call POST /api/2.1/jobs/runs/submit with a run_name and tasks array specifying notebook_task (notebook_path and base_parameters) and the cluster configuration (existing_cluster_id or new_cluster spec)
4. Poll job status by calling GET /api/2.1/jobs/runs/get?run_id={run_id} repeatedly; check result_state in the state object — terminal states are SUCCESS, FAILED, TIMEDOUT, and CANCELED; life_cycle_state of TERMINATED indicates the run has ended
5. Implement a polling loop with exponential backoff (start at 5 seconds, cap at 60 seconds) to avoid hammering the API; Databricks rate limits API calls per workspace
6. Retrieve run output and error details via GET /api/2.1/jobs/runs/get-output?run_id={run_id}; this returns notebook output cells and any error message with a stack trace for failed runs