data-engineering

Write dbt unit tests to validate SQL transformation logic without hitting the warehouse for real data

Query a database over Arrow Flight SQL using the GetFlightInfo/DoGet request flow

Enforce data quality in a Lakeflow Declarative Pipeline (formerly Delta Live Tables) using expectations

Configure Confluent Schema Registry compatibility modes for JSON Schema and Protobuf subjects

Attach severity-aware asset checks to a Dagster asset and control whether failures block downstream materialization

Write SodaCL checks and run a Soda Core scan against a warehouse table

Design a BigQuery materialized view that qualifies for incremental refresh and automatic query rewrite

Configure Airflow 3 DAG bundles to version and source DAGs from multiple repositories

Enforce a dbt model contract on an incremental model without breaking on_schema_change

Author a data contract using the Open Data Contract Standard (ODCS) YAML spec

Trigger and monitor a Census reverse-ETL sync programmatically via its API

Enable Flink buffer debloating to reduce checkpoint alignment time under backpressure

Diagnose and recover from ProducerFencedException in a Kafka exactly-once pipeline

Ingest rows with Snowflake's high-performance Snowpipe Streaming SDK using channels

Achieve exactly-once writes with the BigQuery Storage Write API using a committed-type stream

Configure Prefect 3 task result caching with cache_policy and cache_key_fn

Filter rows during a Debezium ad-hoc incremental snapshot using a signal additional-condition

Chain Snowflake dynamic tables into a DAG using target_lag propagation

Read and write hive-partitioned Parquet datasets in a DuckDB pipeline

Manage a Fivetran connection's schema and table sync config via the REST API

Enable Delta Lake row tracking for stable row IDs across MERGE and UPDATE

Request vended storage credentials from an Iceberg REST catalog when loading a table

Build a custom Airbyte source connector with the low-code Connector Builder YAML manifest

Apply Unity Catalog row filters and column masks to restrict data access

Handle Pulsar message acknowledgment, negative ack, and redelivery

Apply windowing in Apache Beam (FixedWindows, SlidingWindows, Sessions)

Configure Pulsar topic compaction, retention, and TTL

Configure Spark Structured Streaming trigger modes (processingTime, availableNow, continuous)

Choose and use Beam GroupByKey vs Combine.perKey

Apply watermarks and window aggregation in Spark Structured Streaming

Configure checkpointing and recovery in Spark Structured Streaming

Configure Pulsar partitioned topics and message routing modes

Read a Kafka topic into Spark Structured Streaming

Configure Beam triggers and accumulation mode (accumulating vs discarding)

Use Beam side inputs and windowed side inputs

Configure and use Pulsar IO connectors (source and sink)

Implement stream-stream join with watermark in Spark Structured Streaming

Deploy a Dataflow streaming job using a classic or flex template

Configure Dataflow autoscaling and understand Streaming Engine

Implement arbitrary stateful aggregation in Spark Structured Streaming with flatMapGroupsWithState or applyInPandasWithState

Use Pulsar Schema Registry with AVRO and manage schema evolution

Write a stateful Beam DoFn using state and timers

Create Apache Pulsar producers and consumers with all subscription types

Handle Beam watermarks, allowed lateness, and WithTimestamps

Implement Pulsar transactions for exactly-once processing

Choose and apply Spark Structured Streaming output modes (append, update, complete)

Ensure exactly-once in Dataflow and choose between drain and cancel

Use foreachBatch sink in Spark Structured Streaming