Detect and remediate Karpenter node drift caused by AMI or user-data changes

domain: karpenter.sh · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. No action needed to enable Drift itself: as of Karpenter v1.0+ (the current `karpenter.sh/v1` API), Drift is a stable, always-on disruption method with no feature gate to toggle.
  2. For AMI drift, rely on EC2NodeClass spec.amiSelectorTerms (e.g. an `alias: al2023@latest`-style or tag-based selector) — when AWS publishes a new EKS-optimized AMI, EC2NodeClass status.amis resolves to the new AMI id, which no longer matches the AMI baked into already-running NodeClaims.
  3. Detect drift by inspecting the `Drifted` status condition Karpenter adds to affected NodeClaims: run `kubectl get nodeclaims` or `kubectl describe nodeclaim <name>` and look at status.conditions.
  4. For user-data or config changes, edit the EC2NodeClass fields directly (spec.userData, spec.blockDeviceMappings, spec.subnetSelectorTerms, spec.securityGroupSelectorTerms) or the NodePool's spec.template.spec.requirements — Karpenter recomputes a hash of the NodeClaimTemplateSpec/EC2NodeClassSpec and automatically flags non-matching NodeClaims as drifted, no manual annotation required.
  5. Let Karpenter's automated Drift disruption run (it executes before Consolidation in the disruption controller's ordering) to pre-spin replacement nodes, cordon/drain the drifted ones, and terminate them, subject to spec.disruption.budgets rate limits.
  6. To force faster remediation or unblock a stuck rollout, manually delete the affected NodeClaim with `kubectl delete nodeclaim <name>`, and check controller logs / node events for the specific drift reason if replacement isn't proceeding.

Known gotchas

Related routes

Configure Karpenter consolidation and disruption budgets to control node scale-down timing
karpenter.sh · 6 steps · unrated
Use Flux CD drift detection with the Kustomization reconciliation interval and health checks to detect and alert on out-of-band cluster changes
fluxcd.io · 5 steps · unrated
Detect and remediate drift in an AWS CloudFormation stack using the detect-stack-drift API
docs.aws.amazon.com/AWSCloudFormation · 6 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans