{"id":"a623e9c2-90f7-4319-8841-0e5f337730b3","task":"Build an Arrow Flight RPC server in Python to stream large record batches between processes without serialization overhead","domain":"arrow.apache.org","steps":["Subclass pyarrow.flight.FlightServerBase and implement do_get() to serve data for a given Ticket, returning a RecordBatchStream or a GeneratorStream for data that doesn't fit in memory.","Implement do_put() to receive uploaded record batches from a client and persist or otherwise process them server-side.","Implement list_flights() and get_flight_info() so clients can discover available datasets and their schemas before requesting them.","On the client side, connect with pyarrow.flight.FlightClient and use client.get(ticket) to pull a stream of record batches, or client.do_put(...) to push data, avoiding intermediate serialization formats like CSV or JSON.","Add authentication (a handshake-based auth handler or middleware) to the server before exposing it beyond a trusted local network."],"gotchas":["The basic FlightServerBase example transfers data unauthenticated and unencrypted by default; production deployments need TLS plus an auth handler/middleware layered on top.","do_get()'s GeneratorStream must be fed genuinely lazy/iterable batches to handle datasets larger than memory; materializing the full result into a Table before streaming defeats the purpose for large data.","Flight is a lower-level RPC framework, not a query engine; callers needing SQL query semantics over Flight need Arrow Flight SQL specifically, a related but distinct API."],"contributor":"waymark-seed","created":"2026-07-08T17:34:57.823Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/a623e9c2-90f7-4319-8841-0e5f337730b3"}