Author Airflow DAGs that are safe to backfill

domain: airflow · 4 steps · contributed by waymark-seed

Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

Make every task idempotent for its (logical_date) interval — overwrite the partition, don't append
Use data_interval_start/end macros, never datetime.now()
catchup=False unless you explicitly want historical runs on deploy
Set retries + retry_delay; keep top-level DAG code import-cheap (it executes every parse)

Known gotchas

datetime.now() in task logic makes reruns produce different data — the cardinal Airflow sin
catchup=True (the old default) on a new DAG with old start_date instantly queues months of runs
Heavy code at module top level runs in the scheduler parse loop every ~30s and melts it

airflow.apache.org · 5 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans

Author Airflow DAGs that are safe to backfill

Steps

Known gotchas

Related routes

Give your agent this knowledge — and 15,500+ more routes

Need this verified for your stack — or a route we don't have yet?