Decorate task functions with @task and return only JSON-serializable primitives or dataclasses; avoid returning DataFrames or large objects directly
Enable custom XCom backends by setting xcom_backend in airflow.cfg to a class implementing BaseXCom.serialize_value and deserialize_value to store large artifacts in external storage (e.g., S3) and only put the URI into the XCom table
Pass XCom values between tasks using the TaskFlow return value syntax: downstream_task(upstream_task()) so Airflow generates the XCom dependency automatically
Use multiple_outputs=True on a @task decorator when the function returns a dict and you want each key to be an individually addressable XCom key
Set [core] max_active_runs_per_dag and enable_xcom_pickling=False in airflow.cfg (the default in Airflow 3) to enforce JSON-only serialization and surface serialization errors early
Known gotchas
XCom values are stored in the metadata database by default; even with enable_xcom_pickling=False, large JSON payloads bloat the DB and slow scheduler queries — always use a custom backend for objects larger than a few kilobytes
When using a custom XCom backend, the backend's deserialize_value must handle both the legacy format (direct value) and the new URI format, or existing XComs from before the migration will fail to deserialize
Dynamic task mapping with .expand() passes XCom values as mapped arguments; if the upstream task returns a list with thousands of elements, Airflow creates thousands of task instances and can overwhelm the scheduler
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp