{"id":"4f4d1f0e-a60a-4bec-aba3-34b3df56bb3a","task":"Design an idempotent nightly full-vs-delta SIS-to-downstream roster synchronization job with soft-delete handling","domain":"education","steps":["Pull the full source-of-truth roster set (or a delta since the last successful run) keyed by a stable source identifier (e.g. the SIS's immutable student/staff ID), never by mutable fields like email or name.","Stage the incoming batch in a separate table/partition first, then diff it against the downstream system's current active record set by source ID to classify each record as new, changed, unchanged, or missing-from-source.","For records present downstream but absent from the latest full source pull, soft-delete them (set an inactive/deleted flag and timestamp) rather than hard-deleting, so historical references such as grades and enrollments remain valid.","Make each run idempotent by upserting on the source ID with a run/batch identifier and last-synced timestamp, so re-running the same day's job twice produces the same end state rather than duplicates or conflicts.","Run a full reconciliation pass periodically even if delta feeds are used intraday, since deltas alone cannot detect deletions — a record's absence is only knowable by diffing against a complete snapshot.","Log reconciliation summary counts (created/updated/soft-deleted/unchanged) per run so drift between SIS and downstream state is auditable."],"gotchas":["Deltas can tell you what changed but never what disappeared — consuming only delta feeds means a withdrawn student's record silently goes stale; a full snapshot diff is required to catch soft-delete cases.","Hard-deleting every record whose source ID no longer appears is dangerous if the absence was actually a transient SIS export failure (empty/partial file) — guard the sync with a sanity check (e.g. refuse to soft-delete more than X% of records in one run).","Keying joins on mutable fields (name, email, homeroom) instead of the SIS's stable identifier causes duplicate records when a student's identifying info changes mid-year."],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/4f4d1f0e-a60a-4bec-aba3-34b3df56bb3a"}