data-engineering

Apply windowing in Apache Beam (FixedWindows, SlidingWindows, Sessions)

Configure Beam triggers and accumulation mode (accumulating vs discarding)

Handle Beam watermarks, allowed lateness, and WithTimestamps

Write a stateful Beam DoFn using state and timers

Choose and use Beam GroupByKey vs Combine.perKey

Use Beam side inputs and windowed side inputs

Deploy a Dataflow streaming job using a classic or flex template

Configure Dataflow autoscaling and understand Streaming Engine

Ensure exactly-once in Dataflow and choose between drain and cancel

Read a Kafka topic into Spark Structured Streaming

Choose and apply Spark Structured Streaming output modes (append, update, complete)

Apply watermarks and window aggregation in Spark Structured Streaming

Implement stream-stream join with watermark in Spark Structured Streaming

Configure checkpointing and recovery in Spark Structured Streaming

Use foreachBatch sink in Spark Structured Streaming

Configure Spark Structured Streaming trigger modes (processingTime, availableNow, continuous)

Implement arbitrary stateful aggregation in Spark Structured Streaming with flatMapGroupsWithState or applyInPandasWithState

Create Apache Pulsar producers and consumers with all subscription types

Handle Pulsar message acknowledgment, negative ack, and redelivery

Configure and use Pulsar IO connectors (source and sink)

Configure Pulsar topic compaction, retention, and TTL

Use Pulsar Schema Registry with AVRO and manage schema evolution

Implement Pulsar transactions for exactly-once processing

Configure Pulsar partitioned topics and message routing modes