{"id":"15f2edb5-2859-4286-a7cf-829b0e5790c4","task":"Kubeflow Trainer v2: run a distributed PyTorch training job with TrainJob and the torch-distributed runtime","domain":"ml-ops","steps":["Install the Kubeflow Trainer control plane on your cluster and the Kubeflow Python SDK with `pip install -U kubeflow`","Verify the available training runtimes with `for r in TrainerClient().list_runtimes(): print(r.name)` — you should see `torch-distributed` listed","Write a training function containing your full PyTorch DDP/FSDP code with all imports inside the function body (Kubeflow Trainer auto-configures `dist.init_process_group`, `WORLD_SIZE`, `RANK`, and `LOCAL_RANK` via torchrun on every node)","Create the job with `TrainerClient().train(runtime=TrainerClient().get_runtime(\"torch-distributed\"), trainer=CustomTrainer(func=train_fn, num_nodes=N, resources_per_node={\"cpu\":3,\"memory\":\"16Gi\",\"gpu\":1}, packages_to_install=[...]))`, which returns a `job_id`","Check step status with `TrainerClient().get_job(name=job_id).steps` and stream logs with `TrainerClient().get_job_logs(job_id, follow=True)` (or `step=\"node-0\"` for a specific node)","If migrating an existing standalone `PyTorchJob` (kubeflow.org/v1, documented under the Training Operator v1 docs), follow the official Kubeflow Trainer v2 migration guide rather than reusing the old CRD directly"],"gotchas":["All imports needed by the training function must be declared inside the function body itself, since Kubeflow Trainer ships the function to every node for execution","The `torch-distributed` runtime ships with a fixed, pre-installed package set (torch/torchvision/torchaudio); add extra deps per-job via `packages_to_install` on `CustomTrainer`, not by editing the runtime","Kubeflow's own docs describe the standalone PyTorchJob CRD (Training Operator v1) as the prior generation and Kubeflow Trainer v2 (TrainJob) as the current path for new clusters; the two APIs and SDKs are not drop-in compatible, so check the official migration guide before switching an existing setup"],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"verified","url":"https://mcp.waymark.network/r/15f2edb5-2859-4286-a7cf-829b0e5790c4"}