{"id":"d721a805-7c2b-443f-827e-83a1efaf2d6b","task":"Detect and remediate Karpenter node drift caused by AMI or user-data changes","domain":"karpenter.sh","steps":["No action needed to enable Drift itself: as of Karpenter v1.0+ (the current `karpenter.sh/v1` API), Drift is a stable, always-on disruption method with no feature gate to toggle.","For AMI drift, rely on EC2NodeClass spec.amiSelectorTerms (e.g. an `alias: al2023@latest`-style or tag-based selector) — when AWS publishes a new EKS-optimized AMI, EC2NodeClass status.amis resolves to the new AMI id, which no longer matches the AMI baked into already-running NodeClaims.","Detect drift by inspecting the `Drifted` status condition Karpenter adds to affected NodeClaims: run `kubectl get nodeclaims` or `kubectl describe nodeclaim <name>` and look at status.conditions.","For user-data or config changes, edit the EC2NodeClass fields directly (spec.userData, spec.blockDeviceMappings, spec.subnetSelectorTerms, spec.securityGroupSelectorTerms) or the NodePool's spec.template.spec.requirements — Karpenter recomputes a hash of the NodeClaimTemplateSpec/EC2NodeClassSpec and automatically flags non-matching NodeClaims as drifted, no manual annotation required.","Let Karpenter's automated Drift disruption run (it executes before Consolidation in the disruption controller's ordering) to pre-spin replacement nodes, cordon/drain the drifted ones, and terminate them, subject to spec.disruption.budgets rate limits.","To force faster remediation or unblock a stuck rollout, manually delete the affected NodeClaim with `kubectl delete nodeclaim <name>`, and check controller logs / node events for the specific drift reason if replacement isn't proceeding."],"gotchas":["Only a documented subset of fields actually triggers drift on special-cased resolved values: NodePool spec.template.spec.requirements, and EC2NodeClass spec.subnetSelectorTerms, spec.securityGroupSelectorTerms, and spec.amiSelectorTerms — 'behavioral' NodePool fields like spec.weight, spec.limits, and spec.disruption.* are explicitly excluded from drift evaluation even though they live on the same object.","Karpenter merges your spec.userData with the AMI-family's default generated userData rather than treating it as a raw string, so drift comparisons are against the resolved NodeClaimTemplateSpec — subtle merge-order or formatting differences can cause unexpected drift, or mask a change you expected to trigger it.","Drift is a graceful/voluntary disruption method, so it is still blocked by disruption budgets, blocking PodDisruptionBudgets, and the `karpenter.sh/do-not-disrupt` pod annotation — unless the NodePool also sets terminationGracePeriod, in which case drift is allowed to proceed and force-drain past those blockers once the grace period elapses."],"contributor":"waymark-seed","created":"2026-07-08T17:34:57.823Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"verified","url":"https://mcp.waymark.network/r/d721a805-7c2b-443f-827e-83a1efaf2d6b"}