Dev and Staging Kubernetes Costs: What to Cut When Nobody Is Watching

Key Takeaways

• Non-production clusters typically run at under 10% CPU utilization but are provisioned and billed 24/7 – the idle window is 118 out of 168 hours per week.
• Four levers, in order of return: schedule-based shutdown, rightsizing copied-from-production requests, moving workloads to Spot nodes, and dropping HA topology that serves no purpose in dev/staging.
• A CronJob that scales deployments to zero nights and weekends eliminates idle compute for 70%+ of the billing week without changing a single manifest permanently.
• Resource requests copied from production cause 10–16x overprovisioning in dev environments; a 500m CPU request handling a 30m peak means 470m reserved and billed for nothing.
• Orphaned preview environments accumulate silently – automated TTL-based cleanup is the only durable fix.
• Shared staging clusters need guardrails (PDBs, HPA minReplicas, namespace exclusions) before applying any cost optimizations to avoid breaking CI pipelines.

This playbook covers each lever in order of return. You get the arithmetic to prioritize, working configurations to deploy, and a clear-eyed view of what breaks if you move too fast. For the broader picture on Kubernetes cost optimization, that is the pillar post. This one is specifically about what happens between 7pm Friday and 9am Monday.

Why non-production costs what it does

Non-production spend rarely receives the same audit rigor as production. Teams spin up clusters quickly, push manifests from production, and move on. Costs accumulate quietly. Three patterns explain most of it.

Manifests copied from production, including the resource requests

The Cast AI 2026 State of Kubernetes report found 69% CPU overprovisioning across clusters. That number reflects production environments. Non-production clusters are likely worse, because they started as copies of production manifests and nobody adjusted the requests downward. Average CPU utilization across surveyed clusters sits at 8%.

In production, overprovisioning exists as a buffer against traffic spikes. In a development namespace, there are no traffic spikes. A single developer runs integration tests. A CI pipeline runs a build. The 500m CPU request copied from production is handling a workload that peaks at 30m. The remaining 470m are reserved, billed, and doing nothing.

Resource requests in Kubernetes determine node scheduling, not just runtime behavior. Consequently, oversized requests inflate node counts directly. If every pod requests 500m CPU but uses 30m, you are paying for roughly sixteen times more nodes than the workload requires.

Environments that outlive the branch that created them

Feature branch environments are created fast. They are deleted slowly, or not at all. A preview environment for a PR merged two weeks ago and still running costs real money. Multiply that across twenty developers and a medium-sized engineering org, and the orphaned namespaces accumulate into a meaningful billing line.

Most teams have no automated cleanup. Namespaces accumulate. The cluster grows to accommodate them. Eventually someone notices the AWS bill, runs kubectl get namespaces, and discovers eight environments with names like feature-payment-refactor-v3 that nobody has touched in a month.

The fix is structural, not a one-time cleanup script. The time-to-live section below covers the durable approach.

Nobody owns the bill for a namespace nobody uses

Ownership is the underlying problem. In production, someone is on call for costs because production costs affect the business directly. In dev and staging, costs land in a shared cloud bill. No individual developer feels the cost of leaving their environment running over a long weekend.

This is a classic tragedy of the commons. Individual rational behavior, leaving the environment up for convenience, produces a collectively irrational outcome: hundreds of dollars in idle compute per month. Solutions require either automated enforcement or chargeback mechanisms that make the cost visible to individuals. Tools like Kubecost, OpenCost (CNCF), or the cloud provider’s cost explorer with Kubernetes labels enabled all provide namespace-level cost allocation without requiring dedicated infrastructure.

The good news: all four levers in this playbook apply at the infrastructure level. They do not require changing developer behavior, which makes adoption faster and compliance automatic.

Lever 1: Schedule and hibernate

Scheduling is the highest-return lever because the math is straightforward. If a cluster runs 168 hours per week but engineers use it for 50 hours, it sits idle for 118 hours. That is 70% idle time, derived from pure arithmetic: 118 divided by 168 equals 0.702. Every idle hour is a billable hour that produces no value.

For illustrative scale: an EKS cluster running full-time costs roughly $1,382 per month (illustrative figure from DEV Community 2026 FinOps analysis). Running that same cluster only during active working hours, 10 hours per weekday across a 50-hour work week, changes the arithmetic significantly. Base compute: $1,382 x (50/168) is approximately $411 per month. EKS also charges $0.10/hour for the control plane, which is roughly $73 per month regardless of node count, so the floor never drops to zero. Total scheduled cost: approximately $484 per month. That is a 65% reduction from scheduling alone, before touching requests, Spot, or topology. These figures come from a 2026 DEV Community FinOps comparison and serve as directional examples. Your actual savings depend on cluster size and instance types.

What can be shut down overnight and at weekends, and what cannot

Most development workloads can be safely stopped outside business hours. Stateless services restart cleanly. Deployments scale to zero and scale back up without issue. However, some workloads require more planning.

Stateful services, data seeding and warm-up time

Stateful services, such as databases and message queues, lose in-memory state when stopped. That is usually acceptable in development. The more practical issue is data seeding. If your staging environment requires a populated database to be useful, a cold start means waiting for a seed job to complete before the environment is functional. Teams that have not experienced this are sometimes surprised when a Monday morning deploy hangs waiting for Postgres to finish seeding test data.

Mitigation options include running seed jobs as part of the startup sequence, using database snapshots for faster initialization, and keeping staging data volumes persistent even when compute is down. Additionally, factor warm-up time into your schedule: if CI typically needs the environment at 9am, start the cluster at 8:45am to account for pod scheduling and init container time.

One more friction point to address upfront: ArgoCD. If you use GitOps with ArgoCD self-healing enabled, it will fight scale-to-zero. When you scale deployments to zero, ArgoCD sees drift from the desired state and scales them back up automatically. Set spec.syncPolicy.automated.selfHeal: false in your ArgoCD Application spec for non-production apps. Without this, ArgoCD will race the CronJob indefinitely. Plan for this before deploying the CronJob below, or the schedule will appear to work but pods will immediately restart.

For a more targeted approach, use ArgoCD’s ignoreDifferences feature on the Application spec. This tells ArgoCD to ignore replica count drift without disabling self-healing entirely:

ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
      - /spec/replicas

With this configuration, ArgoCD will not override replica counts set by your scale-down CronJob.

Hibernation versus scale-to-zero versus deletion

These three approaches differ in depth and operational impact. Each suits a different situation.

Scale-to-zero sets all deployment replicas to 0 inside the cluster. The nodes remain running. Cost reduction is partial: you stop paying for pod resource reservations, but the nodes themselves continue to incur charges. This is the lowest-risk approach and works well for intra-day pauses or quick validation tests. Here is the native CronJob approach that requires no additional operator or tooling beyond a ServiceAccount with appropriate RBAC:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: scale-down-dev
  namespace: default
spec:
  schedule: "0 19 * * 1-5"
  timeZone: "America/New_York"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: scaler-sa
          containers:
            - name: kubectl
              image: bitnami/kubectl:latest
              imagePullPolicy: IfNotPresent
              command:
                - /bin/sh
                - -c
                - |
                  for ns in development staging preview; do
                    echo "Scaling down namespace: $ns"
                    kubectl scale deployment --all --replicas=0 -n $ns
                  done

This CronJob fires at 7pm Eastern on weekdays and scales all deployments in the development, staging, and preview namespaces to zero replicas. Requires RBAC: create a ServiceAccount (scaler-sa) with permissions to patch Deployments in target namespaces. The timeZone field requires Kubernetes v1.25 or later.

Create a companion scale-up CronJob with schedule: "0 7 * * 1-5". The scale-up command must restore each deployment’s ORIGINAL replica count, not blindly set –replicas=1. Use an annotation to store the original count before scaling down, then restore it on scale-up:

Scale-down (store before zeroing):

for ns in development staging preview; do
  for deploy in $(kubectl get deployment -n $ns -o name); do
    orig=$(kubectl get $deploy -n $ns -o jsonpath='{.spec.replicas}')
    kubectl annotate $deploy -n $ns scaler/original-replicas=$orig --overwrite
    kubectl scale $deploy -n $ns --replicas=0
  done
done

Scale-up (restore from annotation):

for ns in development staging preview; do
  for deploy in $(kubectl get deployment -n $ns -o name); do
    orig=$(kubectl get $deploy -n $ns -o jsonpath='{.metadata.annotations.scaler/original-replicas}')
    kubectl scale $deploy -n $ns --replicas=${orig:-1}
  done
done

The ${orig:-1} fallback handles deployments created after the last scale-down.

The ServiceAccount requires a ClusterRole with patch permissions on Deployments across target namespaces:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: scaler-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: scaler-role
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "deployments/scale"]
    verbs: ["get", "list", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: scaler-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: scaler-role
subjects:
  - kind: ServiceAccount
    name: scaler-sa
    namespace: default

One important conflict to check first: if any Deployment in the target namespace has an HPA attached, scaling to zero replicas will fight the HPA. An HPA with minReplicas: 1 (the default) will immediately scale the Deployment back up. Either remove the HPA from non-production or set minReplicas: 0 – which requires KEDA or Kubernetes 1.23+ with the HPA v2 API. Check for HPAs before enabling the scale-down CronJob: kubectl get hpa -n development.

Cluster hibernation goes further. It scales the cluster itself to zero nodes, eliminating node charges entirely. The control plane (etcd, API server) typically stays active on managed Kubernetes platforms, so resume time is fast. Note: EKS charges $0.10/hour (~$73/month) for the control plane regardless of hibernation, so this is the irreducible floor for any EKS cluster. Cast AI Cluster Hibernation supports EKS, GKE, and AKS, schedules via cron, and runs pre-flight validation on resume to confirm that CNI and DNS are healthy before restoring traffic. Apply the Cast AI Hibernation controller from the Cast AI documentation: pin to a specific release tag rather than the HEAD manifest to ensure reproducibility in GitOps environments.

Deletion removes the cluster entirely. This makes sense for truly ephemeral environments tied to a single PR or experiment. However, deletion means full recreation time on next use, including CNI setup and any cluster-level configuration. For shared staging environments, deletion is usually too disruptive to be a practical recurring pattern.

Working out the real saving: hours down over hours in the week

The saving from scheduling is directly proportional to the fraction of time the cluster is down. Here is the arithmetic for a standard 50-hour work week:

Total hours in a week: 168. Active hours (business hours, Monday through Friday, 10 hours per day): 50. Idle hours: 168 minus 50 equals 118. Idle fraction: 118 divided by 168 equals 70.2%.

Shutting the cluster down for those 118 hours eliminates 70% of compute costs without changing a single manifest. This is schedule math, not measured cluster data. Your actual idle fraction depends on your team’s working hours and time zones. However, for most single-timezone engineering teams, 70% is a reasonable estimate to use for planning.

Teams spanning multiple time zones need to run their own calculation against actual usage patterns. A team with engineers in both London and San Francisco may have a genuinely longer active window, which reduces the idle fraction and lowers the scheduling return. In that case, the other three levers carry more relative weight.

Lever 2: Rightsize non-production properly

Scheduling cuts idle time. Rightsizing cuts the cost of active time. Both are necessary. A cluster that runs only during business hours but still serves 500m CPU requests for workloads that peak at 30m is paying roughly ten times more per active hour than needed.

The requests were inherited, not chosen: 69% overprovisioning applies here too, and worse

The Cast AI 2026 report found 69% CPU overprovisioning across production clusters. Non-production clusters inherited their manifests from those same production environments. Therefore, they carry the same overprovisioning baseline, and usually higher. Production at least sees real traffic that occasionally justifies the buffer. Development sees one engineer running a test or a CI pipeline processing a single build.

The fix is to set non-production resource requests based on actual non-production usage, not production usage. The difference is significant. Here is what that looks like in practice:

Production manifest:

resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
  limits:
    cpu: "2000m"
    memory: "4Gi"

Non-production equivalent:

resources:
  requests:
    cpu: "50m"
    memory: "128Mi"
  limits:
    cpu: "200m"
    memory: "512Mi"

Non-production environments see single-developer or CI pipeline traffic, not real user load. If a pod restarts due to a resource constraint in development, it costs a developer 30 seconds. That is a fundamentally different risk profile from a production restart. As a result, tighter limits in dev and staging are appropriate. Start with conservative requests (50m CPU, 128Mi memory), monitor for OOM kills or CPU throttling, and adjust upward only where evidence requires it.

One important enforcement note: without namespace-level ResourceQuota caps, rightsized requests will be overridden the next time a developer edits their own manifests. ResourceQuota and LimitRange are the Kubernetes-native enforcement layer that makes the new request values stick. Set them once at the namespace level and they prevent individual developers from inadvertently re-inflating requests back to production scale.

Why automated rightsizing suits non-production first: low blast radius, fast feedback

Automated rightsizing observes actual resource usage over time and adjusts requests and limits to match, without manual intervention per workload. Cast AI’s Workload Autoscaler does this continuously. According to Cast AI 2026 data, automated rightsizing cuts provisioned CPUs roughly in half across the cluster fleet.

Non-production is the right place to start automated rightsizing, for two reasons. First, the blast radius of an incorrect recommendation is small. If a limit is set too low and causes an OOM kill, a developer restarts their pod. There is no customer impact. Second, feedback cycles are fast. A change applies, someone runs a test within hours, and the outcome is clear within a day. This makes non-production an effective proving ground before rolling rightsizing to production workloads.

For more on how automated rightsizing works in practice, see the Cast AI Workload Rightsizing deep dive.

Lever 3: Spot by default

Spot instances (AWS Spot, GCP Preemptible, Azure Spot) offer compute at 60-90% below on-demand pricing. They can be reclaimed by the cloud provider with short notice. That interruption risk is why many teams avoid Spot in production. It is also exactly why non-production is the ideal place to adopt Spot nodes by default.

Non-production is where interruption tolerance is highest and adoption is lowest

The irony of Spot adoption is consistent across engineering organizations: the environment most tolerant of interruptions, development, is often the one least likely to use Spot nodes. Production clusters receive the most infrastructure engineering attention, including Spot management tooling. Dev clusters get whatever the initial Terraform configuration specified and never revisited.

Spot interruptions in a development environment cost a developer a reconnect and a pod restart. That is a minor inconvenience. In contrast, Spot savings of 60-90% on compute are substantial. A Cast AI 2025 benchmark found that Spot-heavy configurations save an average of 77% on compute costs. Apply that to the compute portion of your dev cluster bill and the saving is material.

The practical concern is stateful workloads. A Spot interruption terminates the node. If your dev database is on a Spot node without a persistent volume claim or a data checkpoint, you lose data. The solution is straightforward: run stateful workloads on a small on-demand node group (one node is typically sufficient for development) and run all stateless workloads on Spot.

To target Spot nodes in your non-production deployments, add a nodeSelector and toleration:

spec:
  nodeSelector:
    kubernetes.io/lifecycle: spot          # AWS Karpenter label
  tolerations:
    - key: kubernetes.io/lifecycle
      operator: Equal
      value: spot
      effect: NoSchedule

For multi-cloud, use your provider’s equivalent label. On GKE Spot VMs, use cloud.google.com/gke-spot: ‘true’. On Azure Spot, use kubernetes.azure.com/scalesetpriority: spot. Add an on-demand fallback via NodeAffinity with preferredDuringSchedulingIgnoredDuringExecution if interruption tolerance is not total.

Long-running overnight test suites present a separate challenge. If a Spot node is reclaimed mid-run, the test suite may not recover gracefully without checkpointing. Solutions include running integration tests on on-demand nodes, implementing test checkpointing, or simply accepting that an interrupted overnight run will restart in the morning. For most teams, the last option is the pragmatic one given the cost differential.

Cast AI’s Spot management predicts interruptions up to one hour ahead on AWS and up to three hours ahead on GCP. That window is enough time to drain workloads to on-demand nodes before the interruption occurs, eliminating the pod churn that makes unmanaged Spot adoption unreliable. For implementation detail, see the full post on Kubernetes Spot instance optimization.

Lever 4: Drop the production-shaped topology

This lever has the lowest typical saving of the four, but it is also the most invisible cost. Multi-AZ deployments, high replica counts, and over-specified instance types are copied from production and left in place. Nobody challenges them in a non-production environment because nobody owns that namespace or the bill it generates.

Multi-AZ, replica counts, and over-specified node types

Multi-AZ in non-production adds cost without adding value. Cross-AZ data transfer on AWS costs $0.01 per GB, and multi-AZ scheduling requires a minimum number of nodes per availability zone. A three-AZ development deployment that could run on one node is running on three. Furthermore, if an AZ fails in a dev environment, a developer waits a few minutes for a pod restart. That is not worth paying multi-AZ overhead for.

Consolidating to a single AZ eliminates cross-AZ transfer costs and allows the node count to drop proportionally. A representative saving from this change alone is 20-40% on the overall compute bill.

Replica counts in development can safely be set to 1. A production deployment running three or five replicas for redundancy and load distribution needs none of that in a dev environment. One replica per service is sufficient. If a pod crashes, the developer sees the restart. That is actually useful signal, not a problem to prevent.

Check for PodDisruptionBudgets before reducing replicas. A PDB with minAvailable: 1 on a single-replica Deployment will block node drains and prevent Spot interruption handling from completing gracefully. For dev environments running single replicas, either delete the PDB or set minAvailable: 0.

Node types also deserve scrutiny. A development workload running on an m5.4xlarge because that is what production uses is wasteful. Burstable instances (t3, t4g families on AWS) or general-purpose instances at an appropriate size cost significantly less. Dev workloads burst briefly during test runs and idle the rest of the time. Burstable instances are designed for exactly this usage profile.

Note: burstable instances accumulate CPU credits at baseline rate and throttle to baseline performance when credits are exhausted. For CI jobs that run longer than 1-2 hours continuously, a t3.medium will run out of credits and produce slow, throttled builds. Use burstable types for interactive development workloads, not for long-running batch or integration test workloads.

Note: AWS t3 instances launch in Unlimited mode by default since December 2019. In Unlimited mode, the instance charges $0.05 per vCPU-hour for surplus credits rather than throttling. Throttling only occurs in Standard mode when the CPU credit balance hits zero. If your non-production workloads are bursty and short-lived, Unlimited mode (the default) is usually preferable. Switch to Standard mode only if you need a hard CPU ceiling to avoid unexpected charges.

One exception worth noting: shared staging environments that carry significant load test traffic or integration tests requiring multi-service coordination may need more than one replica. Assess the actual traffic pattern before dropping replicas in a shared environment. The rule of thumb is that if a pod restart in the environment would block multiple developers simultaneously, it is not purely development topology and needs a more conservative approach.

The four levers: reference table

Ephemeral and preview environments

Preview environments, per-PR namespaces, and feature branch clusters are valuable for developer experience. They are also a reliable source of unchecked spend. The problem is not creating them. The problem is not deleting them.

Time-to-live as a default, not a cleanup job

Most teams treat cleanup as a periodic manual task: someone runs a script, deletes old namespaces, and the cycle repeats two months later. A more durable approach is to build time-to-live (TTL) enforcement into the environment creation workflow from the start.

kube-janitor provides annotation-based TTL cleanup for namespaces. Add one annotation at creation time and kube-janitor handles deletion automatically. For example:

janitor/ttl: 72h

A 72-hour TTL means any environment created on Monday is gone by Thursday, unless explicitly extended. This is a default, not a ceiling. Teams can annotate long-running environments with a longer TTL when genuinely needed. The key point is that environments expire automatically rather than running until someone remembers to clean them up.

vCluster supports per-PR ephemeral environments that spin up in seconds and can be tied to the lifecycle of the pull request. When the PR merges or closes, the vCluster is destroyed. This makes cleanup structural rather than procedural, eliminating the gap between intent and action that lets orphaned environments accumulate cost.

For teams using GitOps, the environment lifecycle can mirror the branch lifecycle directly. Create a branch, create an environment. Merge or close the PR, destroy the environment. No manual cleanup step, no accumulated cost from forgotten namespaces, no quarterly audit required.

What not to cut

Not all non-production environments are interchangeable. Some exist specifically to reproduce production behavior. Cutting topology cost in those environments undermines the reason they exist.

Environments that exist to reproduce production behaviour

Load testing environments, performance benchmarking clusters, and environments used for security testing need to mirror production topology. If you run a load test against a single-AZ, single-replica environment, the results tell you nothing useful about how production will behave under the same load. Similarly, if you run a security test in an environment with different network topology, you may miss vulnerabilities that only manifest in the production configuration.

For these environments, apply scheduling (shut them down when not in use) but leave topology intact. You can also use Spot instances for the compute if the test can be safely re-run on interruption. However, do not drop replica counts or consolidate to a single AZ when the test result depends on that configuration. The savings from topology simplification are not worth the signal loss.

Shared staging environments that serve as a de facto integration point for multiple teams also need more care. A single-replica shared staging environment means full downtime for every team when that pod restarts. The right question here is whether the environment is truly shared staging or actually an under-resourced production-like environment. If the answer leans toward the latter, the topology cut is the wrong lever. A clearer conversation about environment strategy, combined with proper replica counts for shared services, is the more appropriate path.

Conclusion

Non-production Kubernetes costs accumulate quietly because nobody owns the bill and no customer notices. The four levers address this systematically, in order of return.

Start with scheduling. A 50-hour work week cluster is idle for 118 of 168 hours per week, which is 70% of billable time. Hibernating or scaling to zero during that window eliminates the majority of the cost before touching anything else. For a representative 3-node EKS cluster, that translates to roughly $898 per month in savings, dropping from $1,382 to approximately $484 (including the irreducible $73/month EKS control plane charge).

Next, rightsize the requests. The Cast AI 2026 report found 69% CPU overprovisioning in production clusters. Non-production clusters inherited those same manifests and typically do worse. Automated rightsizing cuts provisioned CPUs roughly in half. The blast radius in non-production is low enough to start immediately without a lengthy change management process.

Then move to Spot. Non-production is the right environment for Spot-by-default, with on-demand fallback for stateful workloads. Spot-heavy configurations save an average of 77% on compute, according to Cast AI 2025 benchmarks.

Finally, drop the production-shaped topology. Single-AZ, replica count of 1, and appropriately sized instance types reduce the remaining cost by another 20-40%.

For the complete framework covering both production and non-production, start with Kubernetes cost optimization.

Cast AI applies all four levers autonomously, without requiring manual configuration for each cluster. Cluster Hibernation handles scheduling. The Workload Autoscaler handles rightsizing. Spot management with interruption prediction handles Spot reliability and reduces pod churn. All of it connects to the same cost visibility layer, so savings are visible and attributable, not just assumed.

Frequently Asked Questions