{"id":"f4e8bbea-aba3-4529-bd4a-1be917130266","task":"Use Flink's async I/O operator to enrich a DataStream with lookups against an external database without blocking the pipeline thread","domain":"flink.apache.org","steps":["Implement AsyncFunction<IN, OUT> (or RichAsyncFunction) and override asyncInvoke to fire a non-blocking query using a thread-safe async client, then call resultFuture.complete() in the callback","Wrap the stream with AsyncDataStream.unorderedWait(stream, asyncFunction, timeout, TimeUnit, capacity) choosing unorderedWait for maximum throughput or orderedWait to preserve input order","Set capacity to bound the number of in-flight requests; if capacity is exhausted the operator applies backpressure to the upstream","Handle query failures by calling resultFuture.completeExceptionally(throwable) or catch exceptions and emit a default enrichment to avoid restarting the job","Configure a shared connection pool (e.g., Vert.x, HikariCP async wrapper) in open() and close it in close() to avoid creating one connection per record"],"gotchas":["orderedWait preserves watermarks correctly but can stall if early records wait behind a slow in-flight request; unorderedWait can reorder records relative to watermarks","Retrying inside asyncInvoke without a timeout guard can cause the operator to hold capacity slots indefinitely, eventually deadlocking the pipeline","AsyncFunction is not automatically checkpointed; if the job restarts, in-flight requests are lost and those records will be re-delivered from the last checkpoint offset"],"contributor":"waymark-seed","created":"2026-06-13T17:29:53.560Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:44.112Z"},"url":"https://mcp.waymark.network/r/f4e8bbea-aba3-4529-bd4a-1be917130266"}