Kubernetes Spot Instances: How to Cut Compute Costs Without Gambling on Reliability

Key takeaways

• Teams running Spot-heavy configurations save 77% on compute on average, compared to 59% for mixed fleets, according to the 2025 Cast AI Kubernetes Cost Benchmark.
• AWS gives you a 2-minute interruption notice. GCP and Azure give approximately 30 seconds (best-effort), which changes how you configure pod shutdown.
• Not every workload belongs on Spot. Stateless services, batch jobs, CI/CD runners, and checkpointed ML training are safe bets. Stateful databases, control-plane components, and payment processors are not.
• PodDisruptionBudgets are the most critical safety mechanism for Spot workloads. Without them, a node drain can take down all replicas simultaneously.
• Cast AI predicts Spot interruptions 1 hour ahead on AWS and up to 3 hours ahead on GCP, then proactively rebalances before the eviction hits.

Why Spot cuts cost so sharply

Cloud providers sell unused compute capacity at steep discounts. When demand increases and they need those resources back, they reclaim the instance with a short interruption notice. That is the entire trade-off. You get compute at 60-90% below on-demand rates in exchange for accepting occasional, predictable-ish eviction.

For workloads designed to survive node loss, this is a reasonable deal. The 2025 Cast AI Kubernetes Cost Benchmark shows clusters running Spot-heavy configurations save 77% on compute. Even mixed fleets (partial Spot coverage) average 59% savings. On a $50k/month EC2 bill, that difference pays for engineering time.

Savings vary by cloud and region

Not all Spot markets behave the same. AWS reprices approximately 197 times per month, so your savings vary constantly. GCP reprices far less often (roughly 0.35 times per month), which makes GCP Spot pricing more predictable. Azure is the most volatile in EU regions: germanywestcentral saw a 150% price increase between 2022 and 2023, australiaeast climbed 131%, and westeurope rose 130%.

Pool size matters too. AWS us-east-1 and us-west-2 have the largest available capacity, which means lower interruption rates. For APAC workloads, ap-southeast-1 (Singapore) and ap-northeast-1 (Tokyo) offer far better availability than newer regions. Avoid af-south-1 and ap-southeast-4 for anything latency-sensitive; AWS Spot Advisor shows both in the highest interruption frequency buckets. On GCP, us-central1 and us-east1 provide the most stable Spot availability globally. For more on EKS-specific cost levers, see the EKS cost optimization guide. For GKE, see the GKE cost optimization guide.

One important GCP distinction: GCP offers two discount VM types. Preemptible VMs use the older API and carry a hard 24-hour maximum runtime limit alongside the 30-second notice. GCP Spot VMs are the current recommendation: no 24-hour limit and the same 30-second best-effort notice. For any new GKE workloads, use Spot VMs, not Preemptible.

Which workloads suit Spot

Safe to run on Spot

Stateless web services recover quickly from interruption, especially when you run at least two replicas and configure readiness probes correctly. Batch jobs and data processing pipelines are natural fits: a terminated job restarts from the last checkpoint with no lasting damage. CI/CD runners are arguably the best Spot use case because each job is ephemeral by design. ML training works on Spot when jobs checkpoint to durable storage every few minutes, so a preemption costs you minutes, not hours.

Not safe for Spot

Stateful databases, including Postgres, MySQL, and Kafka, should stay on on-demand nodes. A forced eviction mid-write can corrupt data or cause replication lag that takes hours to recover. Your Kubernetes control plane (etcd, kube-apiserver) must never run on Spot. Payment processors and anything with strict SLA commitments belong on guaranteed capacity.

Conditional

Some workloads work on Spot with the right architecture. Redis works if it is used purely as a cache (not as a primary store) and you tolerate cold-start latency on reconnection. Kafka consumers work on Spot if partition rebalancing is fast enough for your consumer group SLO. Evaluate your rebalancing time under load before committing.

Interruption handling and fallback

Spot reliability comes from how well your stack handles eviction, not from avoiding it. Two-minute AWS notice sounds generous until you realize your pod shutdown sequence needs to finish within that window. On GCP and Azure with 30-second best-effort notice, the math gets tight. Here is how to build each layer of the stack correctly.

Node-level: AWS Node Termination Handler

If you use Karpenter, you do not need NTH. Karpenter watches for Spot interruption notices via EventBridge and IMDS, cordons the affected node, and begins provisioning a replacement — all natively, without a separate DaemonSet. NTH is for clusters using Cluster Autoscaler or self-managed node groups without Karpenter.

For everyone else, the AWS Node Termination Handler (NTH) picks up the Spot interruption notice, cordons the node, and drains pods gracefully before the instance disappears. The right NTH mode depends on how your nodes are managed:

• Cluster Autoscaler + self-managed node groups: Use NTH IMDS polling mode. It queries the instance metadata service directly, requires no additional AWS infrastructure, and is the simpler setup.
• Cluster Autoscaler + EKS managed node groups: Use NTH Queue Processor mode. This requires both EventBridge and SQS: EventBridge captures the interruption event and routes it to an SQS queue, which NTH then polls. If you run managed node groups and skip the SQS queue, NTH will not receive interruption events for those nodes. The Helm chart for Queue Processor mode requires --set enableSqsTerminationDraining=true along with your queue URL.

Pod-level: grace periods and preStop hooks

The terminationGracePeriodSeconds field belongs at the pod spec level, not inside the container spec. This is a common YAML mistake that causes Kubernetes to silently ignore the value and use the default 30 seconds instead.

The right grace period depends on which cloud you are on. Each cloud has a different interruption notice window, and your preStop hook plus application shutdown must fit within that ceiling.

AWS Spot (2-minute notice)

AWS gives you 2 minutes from interruption notice to instance termination. A 90-second grace period leaves a 5-second preStop hook and 85 seconds for in-flight request draining:

spec:
  terminationGracePeriodSeconds: 90  # Within 2-minute AWS Spot interruption notice
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]

GCP Spot VMs (30s best-effort notice)

GCP Spot VMs provide a 30-second best-effort eviction notice. Keep the total grace period inside that window. A 2-second preStop hook and 23 seconds of application shutdown fits comfortably within the 25-second ceiling:

spec:
  terminationGracePeriodSeconds: 25  # GCP Spot VMs: conservative ceiling within 30s notice
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 2"]

Note: GKE’s managed node shutdown process can give up to 120 seconds in some scenarios, but plan conservatively for the 30-second guaranteed window.

GCP Preemptible VMs and Azure Spot (shorter effective window)

GCP Preemptible VMs and Azure Spot both warrant extra care. Azure Spot gives 30 seconds notice, matching GCP Spot VMs. However, GCP Preemptible VMs have a hard shutdown at approximately 15 seconds from eviction signal, making the effective window shorter than the stated 30 seconds:

spec:
  terminationGracePeriodSeconds: 25  # GCP Preemptible (effective max ~15s); Azure Spot (30s notice)
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 2"]

Note: GCP Preemptible VMs have a hard shutdown at approximately 15 seconds from eviction signal. For Preemptible, shorten preStop to 2 seconds and minimize shutdown logic. For new GKE workloads, use Spot VMs instead to avoid this constraint and remove the 24-hour maximum runtime limit.

PodDisruptionBudgets: the safety mechanism most teams skip

A PodDisruptionBudget (PDB) tells Kubernetes the minimum number of replicas that must stay available during voluntary disruptions, which includes node drains triggered by NTH. Without a PDB, a drain operation can evict every pod in a Deployment simultaneously, taking your service down to zero replicas.

PDBs have two configuration options: minAvailable specifies the minimum number of pods that must stay running, and maxUnavailable specifies the maximum number that can be down at once. For most production services, minAvailable: 1 is the right starting point:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-app

One critical warning: do not set maxUnavailable: 0. This configuration blocks drain operations entirely. If NTH attempts to drain a node before eviction and the PDB blocks all disruptions, the pods will not move before the instance terminates. Use minAvailable: 1 instead, which allows drain to proceed as long as at least one replica stays up.

NTH respects PDBs during the drain phase. When NTH cordons a node and issues eviction requests, Kubernetes checks the PDB before evicting each pod. If evicting the pod would violate the budget, Kubernetes waits until a replacement pod is available elsewhere. This coordination between NTH and PDBs is what makes graceful Spot drains work reliably at scale.

Fallback to on-demand

When Spot capacity disappears, your workload needs somewhere to land. Node affinity with a preferredDuringSchedulingIgnoredDuringExecution rule tells the scheduler to try Spot first and fall back to on-demand if no Spot node is available:

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      preference:
        matchExpressions:
        - key: karpenter.sh/capacity-type
          operator: In
          values:
          - spot

The label karpenter.sh/capacity-type is set by Karpenter on nodes it provisions. If you use Cluster Autoscaler instead, use the node group label defined in your autoscaler node group tags. The label name differs depending on your tooling and how you tag node groups.

Spot with Karpenter

Karpenter changes the Spot game significantly. Rather than pre-defining node groups with specific instance types, Karpenter selects from a broad pool of compatible instances in real time. This means it can automatically pick Spot instance types with lower interruption frequency when higher-frequency types become volatile. For a deeper walkthrough of Karpenter’s Spot mechanics, see the Karpenter Spot instances guide.

The key Karpenter configuration for Spot is the NodePool’s capacity type and instance family diversity. Defining at least 15 compatible instance types in your NodePool gives Karpenter enough flexibility to find available Spot capacity across multiple pools. A minimal NodePool targeting Spot with on-demand fallback looks like this:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot", "on-demand"]
      - key: node.kubernetes.io/instance-type
        operator: In
        values:
        - m5.large
        - m5a.large
        - m5n.large
        - m4.large
        - m6i.large
        - m6a.large
        - m6in.large
        - c5.xlarge
        - c5a.xlarge
        - c5n.xlarge
        - c6i.xlarge
        - c6a.xlarge
        - c6in.xlarge
        - r5.large
        - r6i.large

SpotToSpotConsolidation

Karpenter v0.34.0 (released March 2024) introduced SpotToSpotConsolidation, which allows Karpenter to replace a running Spot node with a cheaper Spot option when one becomes available. This feature is disabled by default and requires two conditions: first, your NodePool must include 15 or more compatible instance types; second, you must enable the feature gate explicitly:

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version <your-karpenter-version> \
  --reuse-values \
  -n kube-system \
  --set controller.featureGates.spotToSpotConsolidation=true

Note: controller.featureGates is the Karpenter v1.x Helm values path. If you run Karpenter v0.x, the path is settings.featureGates.spotToSpotConsolidation. Check your chart version before running.

Without enabling the feature gate, Karpenter will not attempt to consolidate from one Spot type to another, even if a cheaper option appears. With it enabled and sufficient instance diversity in your NodePool, Karpenter continuously optimizes your Spot spend without manual intervention.

How Cast AI handles Spot at scale

Running Spot manually works at small scale. At dozens of node groups across multiple regions and clouds, it becomes a full-time job. The failure modes compound fast: your ML models miss the prediction window and a node evicts mid-batch, on-demand fallback instances land and stay landed because no one triggers rebalancing back to Spot, and your Spot coverage percentage drifts down week over week. Every gap costs money and manual attention.

Three specific problems hit teams at scale:

• Prediction window misses: Spot markets shift faster than reactive tooling can track. By the time an interruption notice arrives, there is no time to stage a replacement gracefully.
• On-demand fallback that stays on-demand: When Spot capacity drops, pods land on on-demand nodes correctly. However, without continuous rebalancing, those on-demand nodes stay running long after Spot capacity recovers. The fallback becomes a permanent cost increase.
• Manual rebalancing across node groups and regions: Rebalancing Spot coverage across multiple availability zones, instance families, and cloud regions requires constant monitoring. At scale, this is not a configuration task — it is a full-time operational responsibility.

Cast AI approaches this as an autonomous infrastructure problem rather than a configuration exercise. The core mechanism is predictive rebalancing: Cast AI’s ML models forecast Spot interruptions 1 hour ahead on AWS and up to 3 hours ahead on GCP, based on pricing signal patterns and capacity trends. When the model predicts an interruption for a node pool, Cast AI proactively provisions replacement capacity and migrates workloads before the eviction arrives. Your pods never wait for a replacement node after an interruption; the replacement is already running when the eviction happens.

Continuous rebalancing runs automatically. On-demand instances that landed as fallback targets get replaced with Spot as soon as availability recovers. Ivan Gusev from OpenX described the outcome directly: “That’s why spot fallback works great for us: we can expect CAST AI to maintain the best possible cost for the cluster by constantly attempting to replace the on-demand capacity with spot.”

Branch, a financial services company, achieved 25%+ EC2 savings through Cast AI’s Spot management and eliminated millions in savings plan commitments that were no longer needed once Spot coverage became reliable. The savings plan lock-in problem is worth calling out: teams that predict they need on-demand capacity often over-commit to savings plans. When Spot reliability improves, those commitments become stranded costs. Cast AI’s approach reduces the need for large on-demand savings plan buffers.

For real-time visibility into Spot availability across instance types and regions before committing to a configuration, the Cast AI Spot Availability Map shows interruption frequency data across all major providers at no cost.

Conclusion

Spot Instances are one of the highest-leverage cost levers available in Kubernetes, but the risk is real without the right safety stack. PodDisruptionBudgets, correct grace period math for your specific cloud and VM type, NTH with the right mode for your node group setup, and on-demand fallback affinity together form a reliable foundation. Start with dev and staging environments to validate your shutdown sequences and PDB configurations before expanding to production workloads. Once the patterns are proven, Cast AI handles the ongoing optimization automatically, predicting interruptions before they hit and keeping Spot coverage high without manual tuning. For a broader view of how Spot fits into a full cloud cost strategy, see the guide to reducing cloud costs with Spot instances.

Frequently Asked Questions