Karpenter Is Running but Costs Haven’t Dropped: A Diagnostic Checklist

Key takeaways

• Oversized requests: Pods request 69% more CPU than they use on average. Karpenter cannot fix what it cannot see.
• Consolidation disabled: consolidateAfter: Never or a missing consolidationPolicy means no bin-packing occurs after initial provisioning.
• NodePool too narrow: Restricting instance families eliminates cheaper alternatives before Karpenter can select them.
• Spot not engaged: Spot is configured but all nodes run on-demand, often due to a known Karpenter fallback limitation (GitHub issue #8889).
• PDBs too strict: maxUnavailable: 0 prevents pod eviction and blocks all consolidation for covered workloads.
• DaemonSet floor: Every node carries 500m to 2 CPU in DaemonSet overhead before any workload pods land.
• Wrong metric: Compute savings exist but are masked by storage, egress, control plane charges, or Reserved Instance accounting.
• Not enough time: Consolidation converges gradually. One week of data is insufficient to judge Karpenter’s impact.

First, confirm Karpenter is actually doing its job

Three things to check before assuming a cost problem

Before diving into cost diagnostics, verify that Karpenter itself is healthy. A misconfigured or crashing controller produces no savings and no obvious signal at the billing layer.

First, confirm the controller pod is running:

kubectl get pods -n kube-system | grep karpenter

If the pod is absent or crash-looping, no provisioning or consolidation will happen. Second, confirm at least one NodePool exists and shows no error conditions:

kubectl get nodepool
kubectl describe nodepool <name>

To confirm Karpenter is actually provisioning nodes, look for Launched and Provisioned events in the kube-system namespace. You can also inspect NodeClaims, the low-level Karpenter v1 objects that track each node request.

# Check Karpenter provisioning events
kubectl get events -n kube-system --field-selector reason=Launched
kubectl get events -n kube-system --field-selector reason=Provisioned

# Check NodeClaims (Karpenter v1 -- lower-level node objects)
kubectl get nodeclaims

If all three checks pass, Karpenter is working correctly. The cost problem is almost certainly upstream, in workload configuration or NodePool settings, rather than in the controller itself.

Cause 1: Your Requests Are the Problem, Not Your Nodes

Provisioned versus requested versus used: 44.87 / 24.9 / 3.94 CPU per hour in one real cluster

This is the most important number in this post. In one production cluster, the three-layer CPU gap looked like this:

• Provisioned: 44.87 CPU/hr, what Karpenter allocated across all nodes
• Requested: 24.9 CPU/hr, what pods declared in resources.requests
• Used: 3.94 CPU/hr, what pods actually consumed at runtime

Provisioned capacity (44.87 CPU/hr) exceeds requested capacity (24.9 CPU/hr) because Karpenter must reserve headroom for DaemonSets running on every node — Datadog, Fluentd, CNI agents, and kube-system components — before workloads can schedule.

Karpenter’s job is to satisfy the middle number, and it does that correctly. The problem is that the middle number is more than six times the actual usage. Across 23,000+ production clusters analyzed in the Cast AI 2026 State of Kubernetes Optimization Report, this pattern is consistent: pods request 69% more CPU than they use, and average cluster-wide CPU utilization sits at just 8%.

No bin-packing algorithm solves this. From Karpenter’s perspective, the capacity is genuinely needed. It sees a pod requesting 4 CPU and provisions a node accordingly. It has no visibility into whether that pod will actually consume 0.3 CPU at runtime. The gap lives in the resources.requests field, not in the scheduler. For a deeper look at how Kubernetes bin-packing interacts with resource requests, that post covers the mechanics in detail.

How to check your own gap in five minutes

The query to run

Run both commands and compare the output side by side:

# Actual CPU usage by pod
kubectl top pods -A --sort-by=cpu | head -20

# Declared CPU requests by pod
kubectl get pods -A -o custom-columns='NAME:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu'

If your top 20 CPU consumers are requesting 5x or 10x what they actually use, the rest of this checklist is secondary. Rightsize the requests first, then revisit the other causes.

The fix: rightsize requests, then let Karpenter re-provision

The open-source path is Kubernetes Vertical Pod Autoscaler in recommendation mode. Run VPA in Off mode for 7 to 14 days to collect usage observations, review the recommendations, then apply them in a controlled rollout. VPA in Auto mode applies changes by restarting pods, which requires compatible PDB settings.

Cast AI’s PrecisionPack automates rightsizing at the workload level, setting requests at p95 actual usage without disruptive pod restarts. It uses Container Live Migration to apply changes with zero downtime. Automated rightsizing reduces wasted compute by approximately 50% (Cast AI 2026 State of Kubernetes Optimization Report). Once requests drop, Karpenter re-provisions to smaller nodes automatically. That is where the savings appear on the bill.

If rightsizing causes OOM kills or throttling in the first week, you can pause VPA recommendations by setting updateMode: Off in the VerticalPodAutoscaler object: kubectl patch vpa <name> -p '{"spec":{"updateMode":"Off"}}' --type merge. This keeps VPA’s learned data while stopping automatic request updates.

Cause 2: Consolidation Is Disabled or Blocked

consolidationPolicy and consolidateAfter

Karpenter v1 supports two consolidationPolicy values: WhenEmpty (evicts only empty nodes) and WhenEmptyOrUnderutilized (evicts underutilized nodes too). The disruption.budgets field (available since Karpenter v1.0) lets you limit simultaneous consolidation events — for example, setting nodes: 0 during business hours prevents any disruption in production.

The consolidateAfter field controls how long Karpenter waits before acting on an underutilized node. Setting it to Never disables consolidation for that NodePool entirely. Also, the timer resets whenever pod activity occurs on the node. High-churn workloads can prevent consolidation from running even when the policy is correct.

Here is a NodePool configured for active consolidation:

# Requires Karpenter v1.0.0+. For v0.x, use consolidation.enabled: true instead of consolidationPolicy.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
      - nodes: "10%"
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]

To confirm whether consolidation is running or blocked, check for Unconsolidatable events:

kubectl get events -n kube-system | grep Unconsolidatable

What blocks consolidation: do-not-disrupt annotations and long grace periods

Even with the correct policy in place, several node-level settings can block consolidation. The karpenter.sh/do-not-disrupt: "true" annotation tells Karpenter to skip all voluntary disruption on that node. This is appropriate for nodes running batch jobs or stateful workloads mid-write, but the annotation is often left on nodes where it was applied temporarily and never cleaned up.

# Block voluntary disruption on a specific node
kubectl annotate node <node-name> karpenter.sh/do-not-disrupt=true

# Remove the annotation to allow consolidation again
kubectl annotate node <node-name> karpenter.sh/do-not-disrupt-

Check for this annotation across running nodes:

kubectl describe node <name> | grep -i disrupt

Long terminationGracePeriod values (600 seconds or more) also slow consolidation, because Karpenter waits for the full grace period before declaring a node safe to remove. For a deeper treatment of consolidation mechanics, the Karpenter consolidation guide covers edge cases and tuning in detail.

Cause 3: NodePool Requirements Are Too Narrow

Restricting instance families removes the cheap options

A NodePool pinned to m5 or c5 instance families cannot select m6i, c6g, or any newer generation that may be cheaper or more readily available in the current AZ. Instead of restricting by family, use instance-category:

requirements:
  - key: karpenter.k8s.aws/instance-category
    operator: In
    values: ["c", "m", "r"]

This opens selection to all current-generation compute, memory, and storage-optimized instance types. Karpenter then picks the cheapest available option that satisfies the pod’s requests across all matching families and generations.

NodePool-level CPU or memory ceilings can also cause premature provisioning halts. If the cluster is approaching a NodePool limit, Karpenter stops launching new nodes. Pods queue as pending, and costs stay flat rather than falling. Review your configured limits against actual cluster usage before assuming a consolidation problem is the root cause.

Cause 4: Spot Is Configured but Never Used

capacityType, fallback behavior, and how to verify what actually ran

Spot Instances save 59 to 77% over on-demand pricing (Cast AI 2026 State of Kubernetes Optimization Report). If your NodePool includes spot as a capacity type but all nodes run on-demand, something interrupted the Spot path.

Check what capacity type your nodes are actually using:

kubectl get nodes -L karpenter.sh/capacity-type

If every node shows on-demand, two things may have happened. Spot was unavailable at launch time and Karpenter correctly fell back. Or, the cluster is affected by GitHub issue #8889: once Karpenter falls back to on-demand, it does not automatically reconsolidate to Spot when capacity returns. Those nodes stay on-demand indefinitely.

The recommended fix is a two-NodePool pattern. Create a primary NodePool that requests Spot capacity only, and a secondary NodePool that requests on-demand at lower weight. Karpenter schedules to the Spot pool first and falls back to on-demand only when necessary. When Spot capacity returns, new workloads land in the Spot pool. Over time, on-demand nodes are consolidated away as the Spot pool absorbs more of the cluster.

# Requires Karpenter v1.0.0+. For v0.x, use consolidation.enabled: true instead of consolidationPolicy.
# NodePool 1: Spot-first (higher weight = preferred)
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-primary
spec:
  weight: 100
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
---
# NodePool 2: On-demand fallback (lower weight = less preferred)
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: on-demand-fallback
spec:
  weight: 1
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]

Higher weight NodePools are tried first. Karpenter falls back to the lower-weight on-demand pool only when Spot is unavailable.

Cause 5: PodDisruptionBudgets Are Too Strict

Karpenter cannot evict a pod when doing so would violate a PodDisruptionBudget. A PDB with maxUnavailable: 0 or minAvailable: 100% means zero pods can be disrupted at any moment. For a single-replica deployment, that prevents all eviction from the node it runs on. Karpenter emits Unconsolidatable events when it encounters this condition, which is the fastest way to confirm the cause.

Check all PDBs across the cluster and look at the ALLOWED DISRUPTIONS column:

kubectl get pdb -A

Any row showing 0 in ALLOWED DISRUPTIONS is blocking consolidation for the pods it covers. For stateless services, maxUnavailable: 1 is the correct setting and allows Karpenter to proceed. Critical services can keep maxUnavailable: 0, but understand that those workloads become consolidation-blocking, which raises the effective node minimum for anything co-scheduled with them.

Cause 6: DaemonSets Set the Floor

Every node pays the DaemonSet tax; on small nodes it dominates

DaemonSets run on every node. Agents like Datadog, Fluentd, and CNI plugins add 500m to 2 CPU of overhead per node, depending on their declared resource requests. Karpenter accounts for this overhead during provisioning. If your DaemonSets collectively require 1.5 CPU and a workload needs 0.5 CPU, Karpenter provisions a node sized for at least 2 CPU, even though the actual workload is small.

On large nodes carrying dozens of workloads, DaemonSet overhead is diluted and becomes a minor percentage of total capacity. On small or lightly loaded nodes, however, DaemonSet cost can represent 50% or more of node capacity. This creates a practical floor on how small any node can be, regardless of how aggressively Karpenter consolidates.

Start the audit here:

kubectl top pods -n kube-system

Compare actual DaemonSet usage to declared requests. If a Fluentd pod requests 500m CPU and uses 20m consistently, that is a rightsizing problem at the DaemonSet level, and fixing it lowers the node floor for the entire cluster. Also avoid scheduling single-replica workloads on their own dedicated small nodes, because the DaemonSet tax makes those nodes expensive relative to the work they perform.

Cause 7: You Are Looking at the Wrong Number

Node cost fell, total bill did not: storage, egress, load balancers, control plane

Karpenter reduces compute cost. It does not touch storage, egress, load balancers, or the EKS control plane. The EKS control plane costs $0.10 per hour, roughly $72 per month, regardless of how many nodes are running. If your compute bill fell by $300 but cross-AZ egress rose by $200 because pod placement shifted after consolidation, the net change looks small on the total bill.

Orphaned EBS volumes, unattached Elastic IPs, and underutilized load balancers also persist after nodes are deleted. These costs accumulate in the background and can offset meaningful compute savings. Use AWS Cost Explorer filtered by service rather than total account spend to isolate what actually changed in the compute line.

Commitment coverage masking the change

If your account uses Reserved Instances or Savings Plans, compute savings from Karpenter may not appear in the bill at all. The RI or SP charge applies regardless of whether the matching instance type runs. When Karpenter removes on-demand nodes that were already covered by a commitment, the savings show up as reduced on-demand overage rather than a lower total. The RI charge continues, and the overall bill moves less than expected.

Segment your Cost Explorer view by purchase option (on-demand, reserved, and Spot) to see where the change landed. The compute savings may already be real, just accounted for differently than expected.

Cause 8: Not Enough Time Has Passed

Consolidation is gradual; what a realistic curve looks like

Karpenter consolidation does not run once and finish. Each consolidation cycle removes or merges a batch of nodes, then waits for the cluster to stabilize before running again. For clusters with many nodes, complex pod topologies, or PDB constraints, convergence takes days, not hours.

Seasonal load patterns also affect the baseline. A cluster measured during a high-traffic period will show different utilization than the same cluster the following week. Thirty days is the minimum window before drawing conclusions about Karpenter’s cost impact. Within that window, monitor consolidation events rather than the bill:

kubectl get events -n kube-system | grep Consolidating

Steady consolidation activity in events, combined with a falling node count, confirms that Karpenter is working. The billing impact follows, typically with a one to two billing cycle lag depending on how AWS aggregates data in Cost Explorer.

A diagnostic order of operations: the checklist

Work through this table in order. Most clusters resolve at Cause 1. The remaining causes are less frequent, but each one can independently prevent savings regardless of how well everything else is configured.

Conclusion

In most cases, Karpenter not reducing costs traces back to Cause 1: pods request far more than they use, Karpenter provisions accurately to meet those requests, and the cluster runs efficiently at the wrong size. The Cast AI 2026 State of Kubernetes Optimization Report found 69% CPU overprovisioning across 23,000+ production clusters. The real-world example above is a concrete illustration: 44.87 CPU provisioned, 24.9 requested, 3.94 actually used. No scheduler recovers that gap without rightsizing the requests first.

Once requests are accurate, the remaining causes in this checklist are each fixable within a day. Karpenter handles provisioning and consolidation well when the inputs reflect reality. For areas where Karpenter has inherent capability gaps, such as pod-level rightsizing, live migration, or multi-cluster cost allocation, see the post on Karpenter’s capability gaps for a full breakdown.

The 44.87 / 24.9 / 3.94 CPU figure is not an anomaly. It is the shape of a production Kubernetes cluster before workload rightsizing runs. Karpenter provisions what it sees — if the requests are wrong, the nodes are wrong. The diagnostic checklist above gives you a sequence of checks, starting with the most common cause and ending with the ones that are easy to miss.

If you want to see the actual CPU gap in your cluster, Cast AI’s cost reports surface that data from a lightweight agent that takes minutes to install. For teams ready to act on it, Cast AI’s free trial applies rightsizing and Spot optimization automatically.

Frequently Asked Questions