{"id":"5ca6375c-4aa8-485b-937e-9f1753ccd002","task":"Read a Kafka topic into Spark Structured Streaming","domain":"data-engineering","steps":["Add the spark-sql-kafka connector dependency matching your Spark and Kafka client versions (e.g., spark-sql-kafka-0-10 artifact; verify the correct artifact coordinates for your Spark version).","Create a streaming DataFrame using spark.readStream.format('kafka').option('kafka.bootstrap.servers', '...').option('subscribe', 'topic-name').option('startingOffsets', 'earliest' or 'latest').load().","The resulting DataFrame has columns: key, value (both binary), topic, partition, offset, timestamp, timestampType. Cast value to string or deserialize as needed.","Apply transformations (parsing, filtering, aggregation) on the streaming DataFrame.","Write results with df.writeStream.format(...).option('checkpointLocation', '...').start() to begin consumption."],"gotchas":["The Kafka connector dependency must match the Spark version exactly; version mismatch causes ClassNotFoundException at runtime.","Kafka credentials (SSL, SASL) are passed via kafka.* prefixed options; do not hard-code secrets — use Spark secrets or external config.","startingOffsets='earliest' on a large topic replays all historical data on a fresh start; use a specific JSON offset map for precise control."],"contributor":"waymark-seed","created":"2026-06-13T14:09:48Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"verification":{"status":"sampled","method":"legacy-file-sample"},"url":"https://mcp.waymark.network/r/5ca6375c-4aa8-485b-937e-9f1753ccd002"}