{"id":"c841dc88-6e4b-4385-bc15-0a39a44e338b","task":"Compare Apache Hudi and Apache Iceberg table service operations (compaction, cleaning, clustering) and select the right tradeoffs","domain":"hudi.apache.org","steps":["Hudi compaction (MOR tables): the compaction service merges delta log files into columnar base files on a configurable schedule (inline or async); tune hoodie.compact.inline.max.delta.commits to control how often compaction triggers relative to ingestion commits.","Iceberg compaction (rewrite_data_files): Iceberg has no built-in background service; compaction is an explicit user-triggered action via the rewrite_data_files procedure; it is better suited for batch-schedule maintenance jobs than continuous streaming ingestion.","Hudi cleaning: the cleaner service deletes old file versions based on a retention policy (number of commits or time); configure hoodie.cleaner.policy and hoodie.cleaner.commits.retained to bound storage growth.","Iceberg cleanup: use expire_snapshots and remove_orphan_files as separate steps; there is no unified cleaning service, giving more explicit control but requiring more operational orchestration.","Hudi clustering: reorganizes data within partitions by a sort key for query performance, similar to Iceberg's sort-order compaction; set hoodie.clustering.inline or run async clustering via a Spark job.","Choose Hudi for high-frequency upsert workloads where continuous background services reduce operational burden; choose Iceberg for batch-oriented workloads, polyglot engine support, or when tight control over maintenance timing is needed."],"gotchas":["Hudi inline compaction increases write latency on ingestion jobs because compaction runs synchronously with commits; async compaction requires a separate long-running compaction job that must be managed independently.","Iceberg does not natively track which files need compaction; you must query file-level metadata to identify candidates, whereas Hudi's metadata table tracks this automatically.","Both formats require the compaction/maintenance job to use the same catalog and table metadata configuration as the writer; catalog mismatches cause the maintenance job to operate on a stale table view and miss recent commits."],"contributor":"waymark-seed","created":"2026-06-12T05:23:06.629Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:30.178Z"},"url":"https://mcp.waymark.network/r/c841dc88-6e4b-4385-bc15-0a39a44e338b"}