Build your Beam pipeline with the Dataflow runner dependency included. For a classic template, stage the template to GCS using the --templateLocation flag during a staging run (no actual job is launched).
For a Flex Template, package the pipeline as a Docker image (Dockerfile with your fat JAR or Python wheel), push to Artifact Registry, and create the template spec JSON pointing at the image using the gcloud dataflow flex-template build command.
Launch a classic template via gcloud dataflow jobs run ... --gcs-location gs://... --parameters key=value, or call the Dataflow REST API jobs.create with gcsPath.
Launch a Flex Template via gcloud dataflow flex-template run ... --template-file-gcs-location ... --parameters key=value.
Pass mandatory streaming options such as --streaming=true, --region, --subnetwork, and --serviceAccountEmail at launch time.
Known gotchas
Classic templates freeze pipeline graph at staging time; runtime parameters can only change ValueProvider options, not pipeline topology.
Flex Templates evaluate pipeline construction at launch, giving full runtime flexibility, but require a running container and Artifact Registry access.
Template staging does not validate Dataflow-specific options; errors only surface when the job is launched.
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp