Karpenter Consolidation and Bin-Packing: How to Cut Node Waste Safely

Sixty-nine percent of Kubernetes clusters over-provision CPU, according to Cast AI’s 2026 State of Kubernetes Optimization report covering more than 23,000 clusters. Average CPU utilization across those autoscaled clusters sits at 8%. Your cluster is almost certainly in that group. Karpenter’s consolidation feature exists to fix this: it repacks running workloads onto fewer nodes, deletes the surplus, and does it continuously without you writing a single cron job. This guide covers how consolidation and bin-packing work in Karpenter v1, which policy fits your risk tolerance, how to protect workloads during disruption, and where native consolidation runs out of runway. If you’re new to Karpenter itself, start with What is Karpenter.

Key takeaways

• 69% of clusters over-provision CPU (Cast AI 2026 State of Kubernetes Optimization report, 23,000+ clusters). Karpenter consolidation is the primary native mechanism for reclaiming that waste continuously.
• Three policies, three risk profiles: WhenEmpty reclaims only empty nodes (most conservative); Balanced scores nodes by consolidation value and targets the best candidates (GA in karpenter.sh/v1 since v1.14.0); WhenEmptyOrUnderutilized evaluates all nodes continuously (most aggressive).
• consolidateAfter applies to every policy, not just WhenEmpty. Set it too low and you’ll churn pods during traffic spikes. Fifteen minutes is the practitioner-tested sweet spot for most production clusters.
• Disruption budgets and the do-not-disrupt annotation are your safety valves. Budgets rate-limit how many nodes Karpenter disrupts concurrently. The annotation protects individual pods from voluntary disruption.
• Stateful workloads need live migration, not just eviction. Draining a node with stateful containers risks data loss or a full restart without the right tooling in place.

What consolidation is and why it cuts cost

Kubernetes schedulers optimize for pod placement at scheduling time. They don’t restructure your cluster after the fact. The result is node fragmentation: as workloads scale down or pods terminate, nodes accumulate idle capacity that the scheduler never reclaims. A cluster running at 70% utilization during peak traffic might drift to 15% overnight while still paying for every node-hour.

Karpenter consolidation runs as a continuous control loop. It evaluates your nodes, identifies where pods can be rescheduled more efficiently, drains the source nodes, and terminates them. No manual intervention required. The cluster moves toward a tighter bin-pack automatically, on whatever schedule your workloads permit.

Karpenter executes consolidation through three mechanisms, evaluated in priority order:

1. Empty Node Consolidation: Karpenter deletes nodes with zero non-daemonset pods in parallel. This is the fastest and safest consolidation path, requiring no eviction.
2. Multi-Node Consolidation: Karpenter deletes two or more nodes simultaneously, optionally launching a single cheaper replacement that fits all the displaced workloads.
3. Single-Node Consolidation: Karpenter deletes one underutilized node, optionally launching a cheaper replacement instance type.

The priority ordering matters operationally. Karpenter always prefers empty nodes before it touches occupied ones. Multi-node consolidation runs before single-node consolidation because collapsing multiple nodes in one action achieves larger savings per disruption event. This hierarchy means your least disruptive wins happen first.

How bin-packing works in Karpenter

Bin-packing is the scheduling problem of fitting workloads onto the minimum number of nodes. For a full treatment of the general concept, see Kubernetes Bin Packing. Karpenter’s approach is specific to its consolidation loop and worth understanding separately.

When Karpenter evaluates a candidate node for consolidation, it simulates rescheduling that node’s pods across the remaining fleet. It checks pod resource requests, node selectors, affinity rules, topology spread constraints, and PodDisruptionBudgets. If all pods fit without violating any constraint, Karpenter proceeds with the disruption. If they don’t fit, the node stays and Karpenter emits an Unconsolidatable event.

When selecting which node to target first, Karpenter prefers nodes with fewer pods, nodes approaching their expiry window, and nodes running lower-priority workloads. This heuristic minimizes blast radius: fewer pods disrupted per consolidation action means fewer rescheduling events for your schedulers to absorb.

One important nuance: Karpenter can replace a node with a cheaper instance type during consolidation, not just delete it. If your NodePool allows a range of instance categories (c, m, and r families, for example), Karpenter might consolidate a half-empty m5.2xlarge and replace it with an m5.large that fits the remaining workload at lower cost. This replacement logic compounds savings beyond what pure deletion achieves.

Consolidation policies

Karpenter v1 ships three consolidation policies. They share the same underlying mechanism but differ in which nodes they consider and how aggressively they act. Choose based on your cluster’s risk tolerance and workload maturity.

WhenEmpty vs WhenEmptyOrUnderutilized

WhenEmpty is the most conservative policy. Karpenter only acts on nodes where every non-daemonset pod has already left. No eviction, no voluntary disruption, no surprises for running workloads. If your team is new to Karpenter consolidation, or your cluster runs latency-sensitive workloads without mature PDB coverage across all critical pods, start here. You’ll capture the low-hanging fruit (truly empty nodes) without any risk to running traffic.

Balanced sits in the middle of the spectrum. It became generally available in karpenter.sh/v1 as of v1.14.0, so it’s production-grade. Balanced scores each node using a consolidation value and targets the highest-value candidates first, meaning it disrupts the nodes where savings are largest relative to risk. You can monitor its decisions with two Prometheus metrics: karpenter_consolidation_score (the computed value per node) and karpenter_consolidation_moves_total (total consolidation moves executed). Balanced works well for clusters where WhenEmpty leaves too much savings behind but WhenEmptyOrUnderutilized feels too aggressive for your current PDB coverage.

WhenEmptyOrUnderutilized is the most aggressive policy. Karpenter evaluates every node continuously. Any node that can be consolidated, will be, subject to your disruption budgets and pod safety constraints. This policy recovers the most cost but requires solid PDB coverage, well-tuned disruption budgets, and a clear understanding of your workload’s tolerance for rescheduling before you enable it in production. For a broader look at how Karpenter handles disruption events beyond consolidation, see Karpenter Disruption and Drift.

consolidateAfter

consolidateAfter is a per-node timer. After a node qualifies for consolidation under the active policy, Karpenter waits this duration before acting. Critically, the timer resets whenever a pod is added to or removed from the node – a node that keeps receiving scheduled work will never fire the timer during that churn period, protecting you from disrupting nodes that are actively serving pods. This makes the timer a stability gate, not just a delay.

Two things to be precise about: consolidateAfter applies to all three policies, not just WhenEmpty. And the reset-on-change behavior is intentional: Karpenter won’t act on a node mid-churn.

The practitioner sweet spot is 15 minutes (900s). That’s short enough to reclaim capacity within a reasonable window after traffic drops, and long enough to avoid consolidating nodes that are temporarily quiet between traffic bursts. If your workloads include longer batch jobs that periodically empty nodes and then refill them, push the timer to 30 minutes to avoid consolidating a node that will need to scale back up shortly. Avoid values under 2 minutes in clusters with high pod churn: you risk disrupting nodes during brief scheduling troughs that resolve on their own.

A consolidation YAML example

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: [c, m, r]
        - key: karpenter.sh/capacity-type
          operator: In
          values: [spot, on-demand]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 15m
    # For testing/demos: reduce to 1-2m. For most production clusters: 15m is the sweet spot.
    budgets:
      - nodes: 10%
      - schedule: "0 9 * * mon-fri"
        duration: 8h
        nodes: "0"
  limits:
    cpu: 1000
    memory: 1000Gi

The two budget entries work together. The first caps concurrent disruptions at 10% of the NodePool’s nodes at any given time. The second freezes all consolidation during business hours, Monday through Friday starting at 09:00, for 8 hours. Karpenter evaluates schedules in UTC – adjust the cron expression for your team’s timezone. Karpenter evaluates budgets and applies the most restrictive match. You get aggressive consolidation overnight and on weekends, with a complete pause during peak traffic hours.

For deeper NodePool configuration beyond consolidation, including instance requirements and limits, see Karpenter NodePools.

Recommended rollout sequence

Don’t enable WhenEmptyOrUnderutilized on day one. The path that works in practice: start with WhenEmpty in a non-production namespace or cluster first. Validate that Karpenter is generating the expected events, that disruption budgets are enforcing your rate limits, and that your PDB coverage is actually blocking eviction where it should. Once that’s confirmed, enable Balanced in production and monitor for a week – check karpenter_nodeclaims_disrupted_total against your baseline. Only move to WhenEmptyOrUnderutilized after you have solid PDB coverage across critical workloads, tested budgets, and Prometheus instrumentation in place to catch unexpected disruption rates.

Note: WhenEmptyOrUnderutilized delivers maximum savings but also the most pod disruption. For most production clusters, Balanced is the right steady state – reserve WEOU for environments with mature PDB coverage and explicit cost-over-stability requirements.

Doing it without downtime

Consolidation moves pods. Moving pods interrupts workloads if you haven’t prepared the right guardrails. Three mechanisms give you precise control over when and how disruption happens: disruption budgets, the do-not-disrupt annotation, and live migration for workloads that can’t tolerate a restart at all.

Disruption budgets

Karpenter’s disruption budgets live in spec.disruption.budgets on your NodePool. They control how many nodes Karpenter disrupts concurrently. They do not control whether individual pods can be evicted; PodDisruptionBudgets handle that. Keep these two concepts separate: disruption budgets are a node-level rate limiter, PDBs are a pod-level eviction gate.

A nodes: 10% budget means Karpenter disrupts at most 10% of your NodePool’s nodes simultaneously. In a 100-node pool, that’s 10 nodes draining at once. In a 10-node pool, it rounds to 1. Use this to cap the blast radius of a consolidation wave, especially when running WhenEmptyOrUnderutilized.

The schedule-based budget in the example above sets nodes: "0" during business hours. This is a complete consolidation freeze: Karpenter will not initiate any disruption while that budget applies. Use it to protect your cluster during peak traffic windows without disabling consolidation entirely for off-peak hours.

Debug consolidation behavior with events: kubectl get events -A --field-selector source=karpenter. The Unconsolidatable event reason tells you when Karpenter evaluated a node for consolidation but couldn’t proceed, whether because a PDB blocked eviction, a budget was active, or pods couldn’t be rescheduled without violating constraints.

Emergency halt: if consolidation causes SLO degradation, set nodes: '0' in your disruption budget immediately – this stops all voluntary disruption. kubectl patch nodepool default --type='json' -p='[{"op": "replace", "path": "/spec/disruption/budgets/0/nodes", "value": "0"}]'

Observability

Before enabling aggressive consolidation, instrument these Prometheus metrics: karpenter_nodeclaims_disrupted_total (ALPHA) – NodeClaims disrupted by consolidation, labeled by reason and NodePool. Spike here tells you disruptions are happening faster than expected; karpenter_nodes_terminated_total (STABLE) gives you a termination rate baseline, alert if this spikes unexpectedly during business hours; karpenter_consolidation_moves_total (ALPHA) is specific to the Balanced policy and counts scored consolidation moves executed. Without these baselines, a runaway consolidation loop and normal overnight activity look identical in logs. Set up a dashboard before you tune budgets, not after.

Start with this alert: rate(karpenter_nodeclaims_disrupted_total[5m]) > 3 — more than 3 NodeClaims disrupted per minute warrants investigation.

do-not-disrupt

Add karpenter.sh/do-not-disrupt: "true" to a pod’s annotations and Karpenter will not voluntarily disrupt the node hosting that pod. The node stays up for as long as that annotated pod runs on it. No eviction, no drain, no consolidation action that would move it.

Good candidates for this annotation:

• Long-running batch jobs where interruption means restarting hours of computation
• Leader pods in distributed systems where re-election carries a latency or consistency cost
• Pods running during a planned freeze window that your scheduled disruption budget didn’t cover

If you set a terminationGracePeriod on your NodePool’s spec.template.spec, Karpenter respects do-not-disrupt and PDBs until that grace period expires, then forcibly terminates the node. This acts as a circuit-breaker for consolidation that gets permanently blocked by a stuck pod. Set it conservatively, long enough for any legitimate batch job to finish, short enough to prevent nodes from becoming permanent fixtures.

Live migration for stateful workloads

Disruption budgets and do-not-disrupt protect pods from being moved at the wrong time. Neither helps when the pod itself can’t tolerate being restarted at all. Stateful workloads with in-memory state, open file handles, or mid-stream processing fall into this category.

Karpenter’s standard drain flow terminates the pod and relies on the scheduler to place a new one. For a stateless API pod, that’s fine. For a database replica, a queue consumer with uncommitted offsets, or an ML training job mid-epoch, it means data loss or a complete restart from the last checkpoint.

Cast AI’s Container Live Migration solves this with CRIU-based checkpointing. The container’s memory state transfers to the destination node before the source terminates. The workload resumes exactly where it left off, with no data loss and no visible restart to the application. This makes consolidation viable for workloads that would otherwise require a permanent do-not-disrupt annotation blocking the node indefinitely. Note: Container Live Migration is currently available for AWS EKS in early access – verify current platform support in Cast AI’s documentation before relying on it in production.

Consolidation limits and when automation helps

Karpenter’s native consolidation handles the straightforward cases reliably: lightly loaded nodes, pods with relaxed affinity rules, clusters with homogeneous workload profiles. It gets harder in production environments that have accumulated real complexity.

Common consolidation blockers in real clusters:

• Topology spread constraints: Pods with topologySpreadConstraints often can’t consolidate onto fewer nodes without violating their spread requirements. Karpenter sees the constraint violation in simulation and leaves the node unconsolidated, emitting an Unconsolidatable event.
• Tight PDBs: A PDB that allows zero disruptions blocks consolidation entirely for the pods it covers. The node stays Ready, it does not go NotReady. But it also doesn’t get consolidated. Karpenter emits Unconsolidatable events repeatedly until the pod terminates or the PDB relaxes.
• Heterogeneous pod resource profiles: Clusters running a mix of memory-heavy and CPU-heavy pods resist efficient bin-packing. The instance types that fit one workload leave dead space for the other, making consolidation less effective than utilization metrics suggest.
• Stateful apps without live migration: Without a way to transfer state, these nodes accumulate do-not-disrupt annotations and gradually become permanent fixtures in your cluster, immune to consolidation.

In a 7-day adversarial EKS benchmark with topology spread constraints, tight PDBs (minAvailable=8), and heterogeneous pod footprints – conditions that specifically stress native consolidation’s limits – Cast AI’s optimization layer saved $302.25 more than Karpenter native WhenEmptyOrUnderutilized alone: a 43% reduction. See the full Karpenter cost optimization methodology and results.

Cast AI extends Karpenter consolidation with three capabilities that address exactly those blockers:

• Evictor: A continuous bin-packing daemon that runs alongside Karpenter and targets the residual fragmentation that WhenEmpty leaves behind. It fills nodes more aggressively than Karpenter’s native loop by understanding pod placement constraints at a workload level.
• Continuous Rebalancer: Workload-aware consolidation that provisions replacement capacity before draining source nodes. This eliminates the scheduling gap between eviction and rescheduling that causes latency spikes in tight clusters with high replica counts.
• Container Live Migration: Covered above. Removes the last blocker for consolidating stateful nodes without service interruption.

If your cluster reaches 30% cost reduction with native Karpenter consolidation and then stalls, the blockers above are almost certainly the cause. Cast AI’s Karpenter optimization is built for that second phase, where native tooling plateaus and workload-aware automation takes over.

Frequently Asked Questions