Configure a Hudi Copy-on-Write table and perform an upsert using record key and precombine field

Steps

1. Add the Hudi Spark bundle JAR to your Spark session and configure: spark.serializer=org.apache.spark.serializer.KryoSerializer and spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension.
2. Write a DataFrame to a new CoW table: df.write.format('hudi').option('hoodie.table.name', 'events').option('hoodie.datasource.write.recordkey.field', 'id').option('hoodie.datasource.write.precombine.field', 'updated_at').option('hoodie.datasource.write.operation', 'upsert').mode('append').save('/path/to/hudi/events').
3. On subsequent writes, use the same upsert operation; Hudi deduplicates by record key, keeping the record with the highest precombine field value when duplicates exist in the incoming batch.
4. Verify the table was created with the correct key configuration by reading back: spark.read.format('hudi').load('/path/to/hudi/events').show().
5. Inspect the Hudi timeline with spark.read.format('hudi').load('/path/to/hudi/events').select('_hoodie_commit_time', '_hoodie_record_key').show() to confirm the metadata fields are present.