Run a preemption-recovering managed spot job on SkyPilot with file mounts for data and checkpoints

Documented steps

1. Write a task YAML with a resources block (accelerators, and use_spot: true for spot capacity), plus workdir, setup and run as with an ordinary task.
2. Add a file_mounts block to stage data: '/remote/path: /local/path' syncs local files, and '/checkpoints: {source: s3://existing-bucket, mode: MOUNT}' mounts a bucket. MOUNT_CACHED is also supported, and 's3://bucket' shorthand works for read-only datasets.
3. Launch it as a managed job: sky jobs launch -n myjob task.yaml. Add --use-spot to request spot explicitly.
4. SkyPilot uploads workdir and any local file_mounts to a cloud bucket so they survive a relaunch. If you do not name a bucket it creates a temporary one (skypilot-filemounts-<user>-<run_id>) and cleans it up when the job finishes.
5. Track state with sky jobs queue, which lists job ids, names, resources, submit time, duration, recovery count and status. Managed jobs do NOT appear in sky status or sky queue.
6. Spot preemptions and hardware failures are auto-recovered: SkyPilot tears down the old temporary cluster, provisions a new one in another region, and restarts the job.
7. Because recovery restarts the job, write checkpoints to a mounted bucket path and resume from the latest checkpoint at the top of your run script. A job that keeps state only on local disk loses it on every preemption.
8. If you also want retries on application errors (non-zero exit), set max_restarts_on_errors in the job's resources block. Only preemptions and hardware failures recover by default.
9. Cancel a managed job using the job id shown by sky jobs queue.
10. Official documentation: https://docs.skypilot.ai/en/latest/examples/managed-jobs.html | https://docs.skypilot.ai/en/latest/reference/yaml-spec.html