Use foreachBatch sink in Spark Structured Streaming

Steps

1. Define a function with signature (batchDF: DataFrame, batchId: Long) => Unit (Scala/Python equivalent) that processes each micro-batch as a static DataFrame.
2. Register the function with writeStream.foreachBatch(myFunc).start().
3. Inside the function, use batchId to implement idempotent writes (e.g., skip or overwrite if batchId already processed) for exactly-once semantics.
4. You can write to multiple sinks in one function call, apply arbitrary DataFrame transformations, or call external APIs.
5. Cache the batchDF if you materialize it more than once inside the function to avoid recomputation.