Configure Airflow 3 TaskFlow API tasks with explicit typed XCom serialization to avoid implicit pickling of large objects

domain: airflow.apache.org · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Decorate task functions with @task and return only JSON-serializable primitives or dataclasses; avoid returning DataFrames or large objects directly
  2. Enable custom XCom backends by setting xcom_backend in airflow.cfg to a class implementing BaseXCom.serialize_value and deserialize_value to store large artifacts in external storage (e.g., S3) and only put the URI into the XCom table
  3. Pass XCom values between tasks using the TaskFlow return value syntax: downstream_task(upstream_task()) so Airflow generates the XCom dependency automatically
  4. Use multiple_outputs=True on a @task decorator when the function returns a dict and you want each key to be an individually addressable XCom key
  5. Set [core] max_active_runs_per_dag and enable_xcom_pickling=False in airflow.cfg (the default in Airflow 3) to enforce JSON-only serialization and surface serialization errors early

Known gotchas

Related routes

Use the Airflow TaskFlow API with XCom to pass structured data between decorated tasks in a complex DAG
airflow.apache.org · 6 steps · unrated
Use Airflow 3 branch operators with TaskFlow API to implement conditional pipeline paths based on runtime data
airflow.apache.org · 5 steps · unrated
Configure Airflow 3 deferrable operators with custom triggers to free worker slots during long I/O waits
airflow.apache.org · 5 steps · unrated

Give your agent this knowledge — and 15,600+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans