Karpenter Disruption and Drift: How to Consolidate Nodes Safely

According to CAST AI’s 2026 Kubernetes Efficiency Report, CPU overprovisioning jumped from 40% to 69% year over year. Karpenter’s disruption system exists to recover that waste. However, without the right guardrails, consolidation can evict workloads at exactly the wrong moment. This guide covers how each disruption method works, how drift detection operates, and which controls keep production clusters stable. For a broader introduction to the scheduler itself, see what is Karpenter.

Key takeaways

• Karpenter disruption covers three mechanisms: consolidation, drift, and expiration. Each has a different trigger condition and a different safety profile.
• Drift detection is event-driven: Karpenter re-evaluates nodes after NodePool or EC2NodeClass changes. In practice, this typically happens within approximately 5 minutes, based on the controller’s re-queue behavior — though this is implementation behavior, not an official SLA.
• Disruption budgets limit concurrent node replacements. The default, when none is defined, is 10% of the NodePool.
• Setting nodes: "0" blocks voluntary disruption. It does NOT block expiration. expireAfter bypasses disruption budgets entirely: the drain starts gracefully (PDBs are respected), but once the NodePool-level terminationGracePeriod elapses, pods are force-deleted regardless of any budget or freeze window.
• The do-not-disrupt annotation on a Pod or Node prevents voluntary disruption. A terminationGracePeriod on the NodePool provides an override after a configurable TTL.
• Use WhenEmptyOrUnderutilized as the consolidation policy in karpenter.sh/v1 for maximum savings. Balanced is a built-in middle ground (no feature gate required). SpotToSpotConsolidation is a separate feature gate that enables replacing a running Spot node with a cheaper Spot alternative — standard Spot node deletion (removing an underutilized Spot node entirely) does not require this gate. Only spot-to-spot replacement does.
• For stateful workloads, combine PodDisruptionBudgets with WhenEmpty policy and AZ-aware scheduling to avoid cross-AZ EBS reattachment failures.

The three disruption methods

Consolidation is the cost-savings engine. Karpenter continuously evaluates whether pods fit onto fewer nodes and removes excess capacity when they do. For a deep dive into how consolidation scoring works, see Karpenter consolidation.

Drift is corrective. When you update a NodePool, Karpenter replaces old nodes that no longer match the new spec. This is how you roll out AMI updates or security group changes without manual intervention.

Expiration is designed for node freshness and security hardening. The key distinction from the other two methods: expireAfter bypasses disruption budgets. When a node’s TTL elapses, Karpenter begins draining it regardless of budget settings. The drain is graceful: Karpenter respects PDBs and pod terminationGracePeriodSeconds. However, once the NodePool-level terminationGracePeriod also elapses, pods are force-deleted. This makes expireAfter unsuitable as a primary rotation mechanism for workloads with strict disruption windows. Rely on drift-triggered AMI rotation instead, and reserve expireAfter for hard compliance TTL requirements.

The three consolidation policies give you a sliding scale of aggressiveness. WhenEmpty removes only nodes that have no pods running, safest for stateful workloads. Balanced considers the same node candidates as WhenEmptyOrUnderutilized but applies a cost-savings-vs-disruption scoring algorithm. Consolidation only proceeds when the savings percentage exceeds the calculated disruption weight for that NodePool. Higher pod priority raises disruption weight, making those nodes less likely candidates. It is a good middle ground for clusters with a mix of latency-sensitive and batch workloads. WhenEmptyOrUnderutilized removes any underutilized node and maximizes cost savings. All three are valid consolidationPolicy enum values in karpenter.sh/v1, with no feature gate required.

What drift is and why it matters

Drift is one of the more powerful features in Karpenter, and also one of the more frequently misunderstood. Here is the core mechanic: when you define a NodePool or EC2NodeClass, Karpenter computes a hash of the fields that affect node behavior. When that hash changes, Karpenter marks existing nodes as drifted and schedules them for replacement.

Drift detection is event-driven: Karpenter re-evaluates nodes after NodePool or EC2NodeClass changes. In practice, this typically happens within approximately 5 minutes, based on the controller’s re-queue behavior, though this is implementation behavior, not an official SLA. Importantly, Karpenter respects disruption budgets during drift-triggered replacements. This makes drift a safer rotation mechanism than expiration for most use cases.

What triggers drift on NodePool

On the NodePool side, drift triggers when you change fields that directly affect the node: nodeClassRef, requirements, taints, startupTaints, and kubelet configuration. Behavioral fields, specifically weight, limits, and anything under disruption.*, do NOT trigger drift. Changing your disruption budget or consolidation policy does not cause a rolling node replacement. That is intentional behavior, not a gap.

What triggers drift on EC2NodeClass

On the EC2NodeClass side, drift triggers for AMI changes, security group changes, subnet changes, and IAM role changes. This is how you implement automated AMI upgrades: update the amiSelectorTerms in your EC2NodeClass and let Karpenter handle the rolling replacement. Pair this with a conservative disruption budget and your nodes rotate safely during off-peak hours.

Notably, Karpenter detects actual AMI changes from AWS, not just spec changes. If Amazon releases a new EKS-optimized AMI and your selector matches it, Karpenter detects that the running AMI no longer matches the selector and marks affected nodes as drifted. You do not need to manually update anything after setting up the selector correctly.

Controlling disruption

Karpenter provides two primary controls for when and how disruption happens: disruption budgets and the do-not-disrupt annotation. Use both. They address different scenarios and operate at different levels of granularity. Disruption budgets work at the pool level, they limit how many nodes across a NodePool can change at once. The do-not-disrupt annotation works at the pod level, it freezes a specific workload without touching the rest of the pool.

Disruption budgets

A disruption budget defines how many nodes Karpenter can disrupt simultaneously within a NodePool. The nodes field accepts either a percentage string like "10%" or an integer like 5. The percentage is relative to the total number of nodes in that NodePool at the time of evaluation, not cluster-wide. On a 10-node NodePool, nodes: "10%" allows 1 node to be disrupted at a time. On smaller NodePools (fewer than 10 nodes), a 10% budget may round down to 0, blocking all voluntary disruption. For small pools, use an absolute integer value like nodes: 1 rather than a percentage. On a 100-node NodePool, the same budget allows 10. When you define multiple budgets, Karpenter applies the most restrictive one at any given time.

Karpenter applies a default disruption budget of 10% when you do not define one explicitly. For large clusters, that represents significant concurrent disruption. Define an explicit budget for every production NodePool.

Budgets also support scheduling via cron expressions in UTC. The duration field defines how long the budget applies. This gives you a clean mechanism to freeze disruption during business hours without manual intervention. Additionally, the reasons field scopes a budget to specific disruption types: Underutilized, Empty, or Drifted. A budget without a reasons field applies to all disruption types.

Disruption budget YAML

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 10m       # 10 minutes is safer than 1m for production
    budgets:
    - nodes: "10%"              # Allow up to 10% of nodes to be disrupted at once
    - nodes: "0"                # Block all disruption during business hours (UTC)
      schedule: "0 9 * * 1-5"  # Monday-Friday 09:00 UTC
      duration: 8h
      reasons:
      - Underutilized
      - Drifted

Note the consolidateAfter: 10m value. Setting this to 1m is too aggressive for production: pods barely have time to reschedule before Karpenter evaluates again. Ten minutes provides a buffer for workload churn without sacrificing meaningful cost recovery.

Also critical: nodes: "0" blocks voluntary disruption during the schedule window. It does NOT block expiration. If your expireAfter TTL fires during your freeze window, Karpenter begins draining those nodes regardless, budget settings do not apply. For this reason, avoid relying on expiration as your only AMI rotation strategy when you have strict maintenance windows.

The do-not-disrupt annotation

The karpenter.sh/do-not-disrupt annotation is a targeted escape hatch for individual workloads. Apply it to a Pod or a Node. When present on a Pod, Karpenter skips voluntary disruption for the node hosting that pod. When applied directly to a Node, the node itself is protected.

Setting the value to "true" gives permanent protection until you remove the annotation. Alternatively, set a duration string like "30m" for time-based protection. This pattern works well for scheduled batch jobs: annotate the pod at launch, and Karpenter leaves the node alone until the job finishes.

One important caveat: terminationGracePeriod on the NodePool overrides do-not-disrupt after a TTL. If a node has been pending replacement longer than the configured period, Karpenter proceeds with disruption regardless of the annotation. This prevents do-not-disrupt from permanently blocking replacement in edge cases where a pod gets stuck.

do-not-disrupt annotation YAML

# On a pod
apiVersion: v1
kind: Pod
metadata:
  name: payment-processor
  annotations:
    karpenter.sh/do-not-disrupt: "true"
---
# On a node
apiVersion: v1
kind: Node
metadata:
  name: ip-10-0-1-42.ec2.internal
  annotations:
    karpenter.sh/do-not-disrupt: "true"

Verifying disruption state

If consolidation appears stalled, check the events first. Karpenter emits Unconsolidatable events when a PDB or do-not-disrupt annotation is blocking a node removal. The NodeClaim shows drift status directly: a Drifted condition means replacement is queued but awaiting a budget window.

# Check NodeClaim disruption conditions (drift, underutilized, expired)
kubectl get nodeclaims -A -o wide

# Watch Karpenter disruption events in real time
# Note: reportingComponent requires Kubernetes 1.25 or later.
# On older clusters, use --field-selector source=karpenter instead.
kubectl get events -A --field-selector reportingComponent=karpenter --sort-by=.lastTimestamp

# Check if consolidation is blocked by a PodDisruptionBudget
kubectl get events -A --field-selector reason=Unconsolidatable

Emergency pause

If consolidation causes an unexpected incident, you need a way to stop all voluntary disruption immediately. Patch the NodePool with a zero-node disruption budget to halt all new voluntary disruption for that pool immediately:

kubectl patch nodepool default --type=merge -p '{"spec":{"disruption":{"budgets":[{"nodes":"0"}]}}}' 

To resume disruption once the incident is resolved, restore the budget to your normal setting:

kubectl patch nodepool default --type=merge -p '{"spec":{"disruption":{"budgets":[{"nodes":"10%"}]}}}' 

Note: Patching to nodes: “0” prevents Karpenter from initiating new voluntary disruption. It does not stop an in-progress drain. It also does not block expireAfter-triggered drains, since expiration bypasses disruption budgets.

Safe patterns for stateful and critical workloads

Stateless workloads are forgiving. A pod that restarts on a new node usually reconnects and resumes within seconds. Stateful workloads are not forgiving. An EBS volume that reattaches to the wrong AZ, a StatefulSet that reschedules mid-transaction, or a cache that warms up cold: these are the scenarios that turn Karpenter disruption from a cost feature into an incident.

PodDisruptionBudgets

PodDisruptionBudgets (PDBs) are the first line of defense. Define minAvailable or maxUnavailable for every StatefulSet and Deployment that runs critical workloads. Karpenter respects PDBs during voluntary disruption: it will not drain a node if doing so violates the PDB. For example, a PDB with minAvailable: 2 on a three-replica StatefulSet ensures Karpenter displaces at most one pod at a time. For additional guidance, see Karpenter best practices.

One common pitfall: setting minAvailable to 100% of a deployment’s replicas, or maxUnavailable: 0, permanently blocks consolidation for that deployment’s nodes. Karpenter will emit Unconsolidatable events but never proceed. If you see nodes that Karpenter never removes despite low utilization, check for zero-tolerance PDBs first. A common fix: set maxUnavailable: 1 for multi-replica deployments and reserve minAvailable: 100% only for single-replica critical services that genuinely cannot tolerate any disruption.

WhenEmpty policy for StatefulSets

For StatefulSets that are difficult to reschedule, consider using consolidationPolicy: WhenEmpty. This policy only removes nodes that are completely empty. Karpenter will not bin-pack StatefulSet pods onto other nodes. Instead, it waits for pods to vacate naturally before reclaiming the node. This approach is more conservative than WhenEmptyOrUnderutilized, but it eliminates unexpected pod migrations for sticky workloads.

EBS and availability zone constraints

EBS volumes with ReadWriteOnce (RWO) access mode bind to a single availability zone. Cross-AZ reattachment fails: a pod cannot bind to the volume if Karpenter moves it to a different AZ. Same-AZ replacement is fast and works correctly.

To keep StatefulSet pods in their original zone, combine a topology spread constraint with node selectors that match the AZ. A practical starting point:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: your-statefulset

When this constraint is in place, Karpenter places replacement nodes in the same AZ as the pod. The RWO volume reattaches successfully.

Therefore, use AZ-specific NodePools or topology spread constraints to keep StatefulSet pods in their original zone. When enabling consolidation on EBS-backed workloads, validate your AZ affinity rules before turning it on in production.

For truly zero-downtime stateful migration, the standard Karpenter disruption path cannot handle the data-movement problem. Container Live Migration addresses this directly by moving the container and its storage together without service interruption.

A critical warning about expireAfter

expireAfter bypasses disruption budgets. When a node’s TTL elapses, Karpenter begins draining it regardless of budget settings. The drain is graceful: Karpenter respects PDBs and pod terminationGracePeriodSeconds during this initial phase. However, once the NodePool-level terminationGracePeriod also elapses, pods are force-deleted. One common misconception: the karpenter.sh/do-not-disrupt annotation on a pod does NOT protect it from expireAfter. Once the NodePool-level terminationGracePeriod elapses, pods annotated with do-not-disrupt will still be force-deleted. The annotation only blocks voluntary disruption, not expiration. If you set expireAfter: 168h on a production NodePool, nodes will rotate every 7 days regardless of cluster state — and freeze windows do not protect them. For stateful workloads, an expiration-triggered replacement during peak traffic can cause serious service disruption even if the initial drain is graceful.

The safer approach: rely on drift-triggered AMI rotation. Update your amiSelectorTerms in EC2NodeClass, set a conservative disruption budget with a maintenance window, and let drift handle replacement on your schedule. Reserve expireAfter for compliance requirements that demand a hard TTL guarantee, and document the budget-bypass behavior explicitly in your runbook.

Scaling disruption management across fleets

Managing disruption budgets on a single cluster is manageable. Managing them across dozens of clusters, with varying workload types, traffic patterns, and change schedules, is where manual tuning breaks down.

Cast AI continuously monitors consolidation efficiency and drift rollout state across your fleet. When consolidation is stalling because a budget window is misaligned with actual low-traffic periods, Cast AI adjusts the window. When AMI drift is queuing up across hundreds of nodes, Cast AI paces the rollout to stay within tolerance. Teams using Cast AI typically see CPU overprovisioning decline from baselines like the 69% industry average — the system continuously corrects for workload churn and demand shifts that would otherwise require weekly manual tuning sessions.

To see how Cast AI applies to your Karpenter fleet, optimize Kubernetes costs with Cast AI.

Frequently Asked Questions