Karpenter Best Practices for Cost, Reliability, and Safe Scaling

Key takeaways

• One of the most important Karpenter best practices is to design multiple, focused NodePools with taints for workload isolation instead of relying on a single catch-all configuration.
• Always enable the SQS interruption queue for Spot. Karpenter does not configure it automatically.
• Set spec.limits.cpu and spec.limits.memory on every NodePool to prevent runaway provisioning.
• Schedule disruption budgets to block consolidation during business hours.
• Use broad instance-category requirements, not explicit type lists, to maximize Spot flexibility.
• Run the Karpenter controller on EKS Fargate or a dedicated node group — never on Karpenter-managed nodes.

NodePool design: focused, not one-size-fits-all

What is Karpenter? It is a Kubernetes node provisioner that replaces the Cluster Autoscaler with faster, more flexible node selection. However, Karpenter’s defaults favor flexibility over safety. In production, that flexibility needs guardrails — starting with NodePool design.

The most common mistake is a single NodePool that matches all workloads. Without isolation, scheduling becomes non-deterministic. Cost attribution becomes impossible, and instance-type policies break silently at scale. Good NodePool configuration separates workloads by tier from the start.

Use multiple NodePools for workload tiers

A practical starting point is two pools: one for stateless workloads using Spot, and one for stateful workloads using on-demand. You can extend this pattern to add GPU pools, team-specific pools, or pools with different CPU architectures (amd64 vs arm64).

Below is a Spot pool for stateless workloads with appropriate limits and disruption settings:

# Spot pool for stateless workloads
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-stateless
spec:
  template:
    spec:
      taints:
        - key: workload-tier
          value: stateless
          effect: NoSchedule
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: 500
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m  # Increase to 5m-10m for production workloads; 1m can cause consolidation churn

Make NodePools mutually exclusive or weighted

When a pod can match multiple NodePools, Karpenter picks one non-deterministically. This is fine for simple clusters. For multi-team clusters or cost attribution requirements, it creates unpredictable behavior that compounds over time.

Two approaches prevent this. First, use distinct taints on each NodePool and matching tolerations on deployments. Second, assign spec.weight values when intentional overlap is needed. Higher weight means higher priority. For example, a spot pool with weight 10 and an on-demand pool with weight 20 routes matching pods to on-demand first when both pools are viable.

Mutual exclusion via taints is simpler to reason about at scale. It also makes cost-per-team attribution clear: each team’s workloads carry tolerations for their assigned pool and nothing else.

Set expireAfter for AMI freshness

Without expireAfter, nodes run until explicitly disrupted or terminated. That means nodes can stay on older AMIs indefinitely, accumulating security and kernel drift. Setting expireAfter: 720h (30 days) ensures rolling replacement without manual intervention.

Warning: expireAfter is forceful — it bypasses disruption budgets. A node that reaches its TTL will be terminated even if your disruption budget specifies nodes: "0". To protect business hours while using expireAfter, use a longer TTL (e.g., 720h minimum) and schedule node refresh through a maintenance window outside business hours.

Stagger expireAfter values across NodePools. If all pools expire at the same time, EC2 API rate limit spikes can cause provisioning delays. A 360h/720h split across pools is a common approach that works well in practice.

Also pin your AMI version in production. Do not use @latest in the EC2NodeClass amiSelectorTerms. During an incident, a @latest alias can pull an untested kernel version into nodes that are spinning up for recovery. Pin to a specific tested alias like al2023@v20240807, then use Karpenter’s drift detection to roll upgrades deliberately from staging to production.

Spot strategy and safe fallback

Karpenter spot instances can substantially reduce node costs. But spot without proper configuration creates reliability risk that outweighs the savings. Three practices separate a production-grade spot setup from a fragile one.

Include spot and on-demand in the same NodePool

When both spot and on-demand are listed in the same NodePool requirements, Karpenter uses price-based scoring to prefer spot. If spot capacity is unavailable in the region and AZ, Karpenter falls back to on-demand automatically. There is no fixed ordering for other capacity types. Karpenter detects spot capacity unavailability during launch and provisions an on-demand node instead. This adds standard node boot time (typically 60-90 seconds), not milliseconds. This pattern prevents workload stalls during regional spot capacity events:

requirements:
  - key: karpenter.sh/capacity-type
    operator: In
    values: ["spot", "on-demand"]
  - key: karpenter.k8s.aws/instance-category
    operator: In
    values: ["c", "m", "r"]
  - key: karpenter.k8s.aws/instance-generation
    operator: Gt
    values: ["2"]

Enable the SQS interruption queue

This is the most frequently skipped configuration step in Karpenter spot deployments. Spot interruption handling is not enabled by default. Without it, Karpenter only learns about a spot reclamation when the node disappears from the Kubernetes API — well past the point where graceful drain is possible.

AWS provides a 2-minute advance notice window via EventBridge for Spot interruptions. To use that window, you need to: create an SQS queue, configure five EventBridge rules (EC2 Spot Instance Interruption Warning, Instance Rebalance Recommendation, Instance State-change Notification, AWS Health Event, and Capacity Reservation Interruption Warning), and pass the queue name to the controller via --interruption-queue. Once configured, Karpenter cordons and drains the node proactively within the 2-minute window.

Additionally, do not run Node Termination Handler (NTH) alongside Karpenter for interruption handling. NTH and Karpenter race to drain the same node. The result is pods left in indeterminate states and double-drain failures. Remove NTH from clusters where Karpenter manages interruption handling.

Maximize instance diversity

Narrow instance-type lists hurt spot availability and block a key optimization. Spot-to-Spot consolidation — replacing a higher-priced Spot instance with a cheaper one — requires at least 15 instance types priced lower than the currently running instance. A list of 3-4 explicit types blocks this entirely.

Spot-to-Spot consolidation is disabled by default. To enable it, set the SpotToSpotConsolidation feature gate to true in the Karpenter controller configuration. Without this feature gate, Karpenter will not attempt to replace a running Spot node with a cheaper Spot node.

Use category and generation constraints instead of explicit instance-type lists. The combination of instance-category: [c, m, r] and instance-generation Gt: 2 opens up dozens of compatible types. This gives Karpenter flexibility to find better capacity pools during regional events and to consolidate Spot-to-Spot when prices diverge.

To exclude specific oversized types that don’t fit your workloads, use node.kubernetes.io/instance-type with operator: NotIn. That keeps the broad pool intact while ruling out specific instance sizes.

Consolidation with disruption budgets

Karpenter consolidation removes underutilized nodes to reduce cost. The default behavior fires consolidation events 24/7 with no awareness of business hours. For most production clusters, that is the wrong default. Read the full guide on Karpenter disruption and drift to understand how the disruption controller works internally.

Choose the right consolidation policy

Karpenter offers three consolidation policies, each with a different risk profile:

• WhenEmpty: removes only empty nodes (only daemonsets running). Most conservative. Use for stateful workloads where pod evictions carry data loss risk.
• Balanced: scores savings against disruption weight and skips marginal consolidations where gains are small. Good middle ground for mixed workloads. (Available in karpenter.sh/v1; check your release notes for the stability status of Balanced in your installed version.)
• WhenEmptyOrUnderutilized: consolidates any node that can be replaced at lower cost. Most aggressive. Best for batch and stateless workloads that tolerate pod restarts.

Apply WhenEmptyOrUnderutilized to stateless pools. Apply WhenEmpty or Balanced to stateful pools. Applying the aggressive policy to StatefulSets without Pod Disruption Budgets in place causes pod evictions without warning and can result in data corruption.

Use disruption budgets to protect business hours

The default Karpenter disruption budget allows up to 10% of managed nodes to be disrupted simultaneously. That budget has no schedule, so it applies equally at 3 AM and 11 AM on a Monday. Consolidation firing during peak traffic causes mid-day pod restarts that often look like application errors until you trace the event timeline.

The fix is a scheduled budget that sets nodes: "0" during business hours. The configuration below allows off-hours consolidation while freezing all voluntary disruptions during a Monday-to-Friday 9-to-5 UTC window:

disruption:
  consolidationPolicy: WhenEmptyOrUnderutilized
  consolidateAfter: 1m  # Increase to 5m-10m for production workloads; 1m can cause consolidation churn
  budgets:
    - nodes: "10%"
    - nodes: "0"
      schedule: "0 9 * * MON-FRI"
      duration: 8h

Budget schedules run in UTC. Adjust cron expressions for your team’s actual business hours. You can also use the optional reasons field to scope a budget to specific disruption types — for example, freezing consolidation while still allowing drift-triggered rolling updates to proceed.

Limits and guardrails

Without resource limits, a misconfigured deployment or runaway Horizontal Pod Autoscaler can provision hundreds of nodes before any billing alert fires. NodePool limits are provisioning guardrails that operate independently of cloud cost alerts — they stop the provisioning loop at the Karpenter level.

Set both spec.limits.cpu and spec.limits.memory on every NodePool. Size them at 110-120% of expected peak load. That headroom prevents artificial ceilings during normal burst conditions while still catching runaway provisioning. When a NodePool hits its limit, Karpenter stops provisioning and pending pods wait until capacity frees up.

One important caveat: limit checking is eventually consistent. During fast scale-outs, provisioning can briefly exceed the limit before Karpenter reconciles. This is expected behavior. Size your limits with that brief overage in mind.

Monitor for extended Pending pods as a signal that a NodePool has hit its limit. Alert on pods stuck in Pending for more than 5 minutes. Without this alert, a limit hit looks indistinguishable from a generic scheduling failure until you check Karpenter’s logs directly.

Labels, taints, and placement control

Labels and taints are the primary mechanism for controlling which pods land on which nodes. Karpenter respects standard Kubernetes scheduling constraints — node selectors, affinities, topology spread constraints — and applies them at provisioning time, not just at scheduling time.

Taints on NodePools prevent pods without matching tolerations from landing on those nodes. This is how workload tier separation is enforced. A spot pool with a workload-tier: stateless NoSchedule taint only accepts pods that explicitly tolerate it. Without taints, scheduling on overlapping pools is non-deterministic.

Well-known labels reference

Use these labels in NodePool requirements and pod nodeSelector or affinity rules to control placement precisely:

EC2 Capacity Reservations are configured via capacityReservationSelectorTerms in EC2NodeClass — they are not a karpenter.sh/capacity-type value.

A few labels need extra care. Availability zone names are account-specific: us-east-1a in one AWS account maps to a different physical AZ than us-east-1a in another. Where possible, use topology spread constraints with topologyKey: topology.kubernetes.io/zone rather than pinning to specific zone names.

The karpenter.sh/do-not-disrupt: "true" annotation protects individual pods from voluntary disruption (consolidation and drift). Use it on batch jobs and ML training runs that cannot be safely interrupted mid-execution. However, avoid placing it on more than 20-30% of total pods. Over-annotation degrades consolidation efficiency to the point where Spot savings no longer justify the overhead.

Observability and cost visibility

Running Karpenter without observability means problems accumulate invisibly. The key metrics span across stability tiers (stable, beta, alpha), and knowing which ones to alert on versus which ones to use for debugging saves significant investigation time.

These metrics cover day-to-day operations effectively:

• karpenter_nodes_created_total: Track provisioning rate by NodePool and zone. An unexpected spike often points to a runaway HPA or missing NodePool limits.
• karpenter_nodes_terminated_total: High churn — creates and terminations at similar rates — signals consolidation thrashing. Tuning consolidateAfter upward usually resolves this.
• karpenter_nodeclaims_disrupted_total: Shows disruptions by reason (drift, consolidation, expiry). Use it to verify that scheduled budgets actually suppress disruptions during protected windows.
• karpenter_pods_state: Alert on pods in Pending for more than 5 minutes. This often signals a limit hit or an unsatisfiable scheduling constraint.
• karpenter_pods_startup_duration_seconds: Track P99 startup time. A regression after a config change often points to AMI issues or node init problems.

Beyond metrics, check node events. When Karpenter cannot consolidate a node, it emits an Unconsolidatable event with a specific reason. Common reasons include a PDB blocking pod eviction, a preferred anti-affinity preventing rescheduling, or no cheaper replacement node available in the current market. These events are the fastest path to finding stuck nodes that should have been removed but weren’t.

Karpenter has no built-in cost attribution per namespace or team. For cost visibility at the workload level, a separate tool is required. Without namespace-level cost tracking, you cannot identify which team caused a sudden cost spike, making chargeback and optimization conversations difficult.

Debugging NodePool issues

When a NodePool is not behaving as expected, these commands surface the most common root causes quickly:

# Check Karpenter controller logs for scheduling decisions
kubectl logs -n karpenter deployment/karpenter --since=10m | grep -E 'ERROR|WARN|scheduling|provisioning'

Controller logs reveal why nodes weren’t provisioned or consolidated — check them first before inspecting NodeClaims or events.

# List all Karpenter-managed nodes and their NodePool
kubectl get nodes -l karpenter.sh/nodepool

# Inspect a NodeClaim for provisioning status
kubectl get nodeclaims
kubectl describe nodeclaim <name>

# Find Unconsolidatable events and NodePool scheduling issues
kubectl get events -A --field-selector source=karpenter --sort-by=.lastTimestamp | tail -20

Anti-patterns to avoid

Most production incidents with Karpenter trace back to a predictable set of mistakes. The table below captures the highest-impact ones:

How Cast AI closes the gaps Karpenter leaves open

Karpenter is excellent at provisioning right-sized nodes based on pod resource requests. The structural problem is that pod requests are often wrong. According to the Cast AI 2026 State of Kubernetes Optimization Report (covering 23,000+ clusters), 69% of Kubernetes clusters overprovision CPU, and average CPU utilization across autoscaled clusters sits at just 8%. Karpenter provisions a node sized for what pods ask for, not what they actually use.

This gap cannot be closed at the node level alone. Fixing it requires rightsizing at the pod level — adjusting CPU and memory requests based on real consumption data.

Karpenter Optimization with Cast AI adds three capabilities that work alongside Karpenter’s node provisioning:

• Workload rightsizing: Cast AI observes actual CPU and memory consumption per workload and adjusts resource requests automatically. Karpenter then receives accurate requirements, which enables tighter bin-packing and reduces the need for oversized nodes.
• Spot interruption prediction: Rather than reacting to the 2-minute EC2 notice, Cast AI uses ML-based prediction to identify at-risk Spot nodes and replace them proactively before AWS reclaims them.
• Container live migration (CRIU): Cast AI moves running containers between nodes without restarts, including stateful workloads backed by persistent storage. This makes consolidation possible for workloads that Karpenter would otherwise leave in place.

Together, these capabilities address the structural gap between “Karpenter is provisioning nodes” and “my cluster is actually efficient.” Node autoscaling and pod rightsizing solve different parts of the same cost problem.

Frequently Asked Questions