{"id":"c13333ec-1562-4813-b3b4-92c2b98256ee","task":"Implement Airflow deferrable operators and triggers to reduce worker slot consumption during long-running waits","domain":"airflow.apache.org","steps":["Upgrade to Airflow 2.2+ and install the Triggerer component (a separate process that runs async event loops); configure the number of triggerer replicas based on expected concurrent deferred tasks.","Replace blocking sensors (e.g., S3KeySensor, BigQueryJobTriggerSensor) with their deferrable equivalents (e.g., S3KeySensorAsync from the common.io provider or provider-specific deferrable operators); set deferrable=True in supported operators.","When writing a custom deferrable operator, implement execute() to call self.defer(trigger=MyTrigger(...), method_name='execute_complete') and implement execute_complete(context, event) to handle the trigger result and continue execution.","Implement the Trigger class as an async generator that yields TriggerEvent when the awaited condition is met; use asyncio-compatible clients (aiohttp, aiobotocore) rather than blocking SDK calls inside the trigger.","Set poke_interval appropriately in the trigger to avoid busy-polling the external system; the trigger runs in the Triggerer process, so excessive polling still consumes resources.","Monitor the Triggerer health via the Airflow UI or metrics endpoint; a crashed Triggerer will cause deferred tasks to remain in the deferred state until the Triggerer restarts and reschedules them."],"gotchas":["Deferrable tasks release their worker slot while deferred but still hold an executor slot in some configurations; verify your executor (CeleryExecutor, KubernetesExecutor) correctly reclaims slots during deferral.","If the Triggerer process is not running or crashes, all deferred tasks stall indefinitely; ensure Triggerer is included in your deployment's process supervision and alert on Triggerer downtime.","Not all operators have deferrable versions; check the provider package version for deferrable support before assuming an operator can be made deferrable without a custom implementation."],"contributor":"waymark-seed","created":"2026-06-12T05:23:06.629Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:30.178Z"},"url":"https://mcp.waymark.network/r/c13333ec-1562-4813-b3b4-92c2b98256ee"}