Kubeflow Trainer v2: run a distributed PyTorch training job with TrainJob and the torch-distributed runtime

Steps

1. Install the Kubeflow Trainer control plane on your cluster and the Kubeflow Python SDK with `pip install -U kubeflow`
2. Verify the available training runtimes with `for r in TrainerClient().list_runtimes(): print(r.name)` — you should see `torch-distributed` listed
3. Write a training function containing your full PyTorch DDP/FSDP code with all imports inside the function body (Kubeflow Trainer auto-configures `dist.init_process_group`, `WORLD_SIZE`, `RANK`, and `LOCAL_RANK` via torchrun on every node)
4. Create the job with `TrainerClient().train(runtime=TrainerClient().get_runtime("torch-distributed"), trainer=CustomTrainer(func=train_fn, num_nodes=N, resources_per_node={"cpu":3,"memory":"16Gi","gpu":1}, packages_to_install=[...]))`, which returns a `job_id`
5. Check step status with `TrainerClient().get_job(name=job_id).steps` and stream logs with `TrainerClient().get_job_logs(job_id, follow=True)` (or `step="node-0"` for a specific node)
6. If migrating an existing standalone `PyTorchJob` (kubeflow.org/v1, documented under the Training Operator v1 docs), follow the official Kubeflow Trainer v2 migration guide rather than reusing the old CRD directly