How Cast AI’s Automation Decides: The Guardrails, Rollbacks and Evidence Behind Each Action

The hesitation most platform engineering teams have before enabling Kubernetes automation is not about the technology. It is about control. When automation makes a decision in your cluster, you want to know what it looked at, why it acted, what would stop it from doing something destructive, and what happens when it gets it wrong. Those are reasonable questions, and they deserve precise answers.

According to Cast AI’s 2026 State of Kubernetes Optimization Report, which analyzed tens of thousands of clusters across AWS, GCP, and Azure, 69% of clusters overprovision CPU, and average CPU utilization on autoscaled clusters sits at just 8%. That waste exists even with autoscaling enabled. Node autoscaling solves for capacity headroom. It does not solve for pod-level request accuracy. Fixing that second problem is where automation needs to go deeper, and where the trust question becomes most acute.

Key Takeaways

• Cast AI reads SLO signals, not just utilization counters, before every change: error rate, p99 latency, OOMKill rate, and PSI (Pressure Stall Information) are all inputs to the decision engine.
• Guardrails are pre-flight checks, not post-hoc recovery. Rebalancing aborts before nodes are drained if the configured savings threshold cannot be met. PDB violations cause immediate revert to active node state.
• Every action is logged with a timestamp, operation name, and initiator (user or policy name), and is exportable as YAML or JSON via the audit log API.
• Recommend-only mode is a documented onboarding path. You observe recommendations, validate them against your expectations, and then enable automation on your timeline.
• In one representative cluster, OOMKills dropped from 40 to 50 per measurement window (typically 24 hours) to near zero within days of enabling automated rightsizing, while provisioned CPU dropped approximately 50% simultaneously.
• Karpenter and native VPA do not have equivalent guardrails. Cast AI’s savings threshold pre-flight, PDB revert logic, and HPA conflict resolution fill the gap those tools leave open.

How Cast AI Reads Your Cluster

Before Cast AI takes any action, it gathers signals. The inputs are broader than most teams expect.

For workload rightsizing, the Workload Autoscaler monitors application health signals alongside raw resource metrics. Error rate, p99 latency, and OOMKill rate are tracked per workload. PSI (Pressure Stall Information) metrics provide CPU contention awareness that raw utilization does not capture. A pod can show 70% CPU utilization while also experiencing significant throttling because the measurement window averages over bursts. PSI stall detection catches that. PSI is available on Linux kernel 4.20+ and is not present on all managed node OS images — verify PSI availability on your node image before relying on this signal. The startup metrics exclusion feature also matters here: Cast AI ignores resource consumption during the pod initialization phase to prevent recommendations that would over-provision or under-provision based on startup spikes that are not representative of steady-state behavior.

For node autoscaling and rebalancing, Cast AI reads unschedulable pod events and maps them against available instance inventory. It also ingests real-time Spot market pricing and interruption prediction data. The Spot Handler DaemonSet runs on each node as a monitoring-only component. It does not drain nodes or reschedule pods directly. It reports interruption signals to Cast AI’s ML models, which make the actual decision. For AWS, Cast AI’s ML models predict interruptions approximately one hour ahead of the cloud provider’s two-minute Spot warning. For GCP, the ML-based prediction horizon extends to approximately three hours ahead. These are Cast AI’s ML-based prediction horizons, not cloud provider guarantees — Spot warnings from the provider itself arrive 30 seconds to 2 minutes before actual interruption. That difference between a one-hour prediction window and a two-minute panic window is what makes the difference between a graceful planned migration and a disruptive emergency drain.

The HPA interaction is where Cast AI’s decision logic gets most nuanced. The Workload Autoscaler defers to HPA as the first line of defense for traffic spikes. Three distinct behaviors govern this relationship:

Downward rightsizing (cost savings):

Reclaiming slack CPU and memory from overprovisioned workloads runs continuously regardless of HPA state. HPA is not involved in this path.

Upward rightsizing (OOMKill prevention):

Adding request headroom triggers specifically when HPA is at its maxReplicas limit and the workload is still resource-constrained. HPA cannot prevent OOMKills by adding more pods when it has already hit its ceiling. Vertical adjustment is the only lever left.

HPA converter (preventing scaling drift):

Whenever Cast AI adjusts CPU or memory requests in either direction, it runs the HPA converter, which translates percentage-based HPA utilization targets to equivalent absolute values. This prevents a scenario where you rightsize a deployment’s CPU requests downward and HPA starts firing more aggressively because the utilization percentage calculation shifted, even though actual load did not change.

The signal picture before any action includes: resource utilization history over a configurable look-back period, SLO health signals (error rate, latency, OOMKill rate, PSI), HPA state and capacity, unschedulable pod events, Spot pricing and interruption forecasts, and the configured policy constraints for that workload or node template. That full picture is what separates a decision that considers application health from a naive cut based on a rolling average.

Kubernetes Automation Guardrails

Guardrails in Cast AI are not recovery mechanisms. They are constraints enforced before an action executes. The distinction matters because recovery after a bad action means you already had an incident. Pre-flight enforcement means the bad action never happens.

Node template constraints. Every Cast AI autoscaler action operates within the bounds of your node templates. Node templates define instance type filters, CPU and memory minimums and maximums, storage requirements, GPU requirements, Spot versus On-Demand preferences, and availability zone targeting. The autoscaler will not provision nodes beyond the CPU limit you configure for unscheduled pods. That limit is also respected as the final state after rebalancing, though temporary exceedance during a rebalancing run is allowed and documented. If the inventory to satisfy your constraints is not available, the autoscaler waits rather than violating the template.

PDB enforcement with automatic revert. When Cast AI drains a node during rebalancing or Spot migration, it respects PodDisruptionBudgets. If a drain attempt fails because evicting a pod would violate the PDB, Cast AI does not force the eviction. It reverts the node to active state. The failed drain is annotated on the node with rebalancing.cast.ai/status=drain-failed so you can see exactly what happened and why. The graceful eviction timeout is configurable with a default of 20 minutes and a maximum of 180 minutes. PDB-blocked drains are surfaced in the audit log and can trigger notifications via PagerDuty or Slack webhooks.

Savings threshold pre-flight for rebalancing. Scheduled and on-demand rebalancing operations have a configurable minimum savings threshold. Before Cast AI starts draining original nodes, it runs a pre-flight check to verify the projected savings meet that threshold. If cloud provider inventory gaps or pricing shifts mean the threshold cannot be achieved, the rebalancing operation aborts before any disruption occurs. This is not a rollback. The original nodes were never touched. The pre-flight check fires before any drain begins.

Per-workload exclusion annotations. Stateful services, JVM workloads without explicit heap bounds, workloads with hard SLO commitments, or any deployment currently under active investigation can be excluded from automated rightsizing via annotation. This is a hard boundary. The Workload Autoscaler will continue generating recommendations for excluded workloads in recommend-only mode, but it will not apply changes. To exclude a workload, add the following annotation to its pod template:

# Exclude a workload from automated rightsizing
# Verify the current annotation key at docs.cast.ai/docs/workload-autoscaler
annotations:
  scheduling.cast.ai/paused: "true"

The annotation-based approach means exclusions are managed as code alongside the workload definition, not buried in a UI configuration page.

Whitelisting mode for controlled expansion. Available from Workload Autoscaler v0.58.3, whitelisting mode inverts the default behavior. Instead of automating all eligible workloads and relying on exclusion annotations to carve out exceptions, you explicitly opt workloads into automation. To opt a workload in when running whitelisting mode, add the corresponding opt-in annotation:

# Opt a workload into automation when whitelisting mode is active
# Verify the current annotation key at docs.cast.ai/docs/workload-autoscaler
annotations:
  cast.ai/vpa-enabled: "true"

This is useful for teams that want to expand automation incrementally, starting with non-critical workloads, verifying behavior over a period of time, and then expanding the scope. The mental model shifts from “everything automated unless annotated out” to “nothing automated unless annotated in.”

Min and max node count bounds. The autoscaler respects hard minimum and maximum node counts configured per node group. It will not scale below the minimum even if workload demand would allow it, and it will not scale above the maximum regardless of unschedulable pod queue depth. These bounds are the outermost layer of the guardrail stack. Everything else operates within them.

Workload Autoscaler high availability. The Workload Autoscaler itself runs in high-availability mode with two replicas since v0.30.0. Replicas are scheduled with anti-affinity to ensure they run on different nodes. One replica performs operations and one is standby. The automation control plane does not become a single point of failure in your cluster.

Human-in-the-loop escalation. Actions that fall outside established policy guardrail boundaries require human approval before execution. Cast AI’s autonomous operation applies only to changes that fit within your configured policies. For changes at the boundary or beyond it, the system escalates rather than acting unilaterally. The distinction between automated and autonomous Kubernetes operations is meaningful: automation executes a predefined sequence, while autonomy means the system makes decisions within guardrails, with escalation paths for edge cases.

When Automation Gets It Wrong

No automation system is error-free. The question is whether the failure modes are bounded and recoverable. Cast AI’s failure handling works at several levels.

Emergency automation pause. When something slips through, the first tool is a full stop. You can pause all Cast AI automation for a cluster from the console or via API. This is the break-glass option: it stops all autonomous actions immediately while existing workloads continue running. New automated actions will not initiate until automation is re-enabled. The API path for emergency pause follows the cluster policies endpoint:

# Pause all Cast AI automation for a cluster
# Full API reference: https://docs.cast.ai/reference
curl -X PUT \
  "https://api.cast.ai/v1/kubernetes/clusters/YOUR_CLUSTER_ID/policies" \
  -H "X-API-Key: YOUR_CAST_AI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

Use emergency pause during major deployments, incident investigations, or any situation where you need full manual control.

HPA ownership snapshot restore. When the Workload Autoscaler takes ownership of an HPA-managed deployment, it preserves a snapshot of the original HPA configuration. If you need to revert Cast AI’s HPA management on a workload, setting takeOwnership=false restores the HPA from that snapshot. This is available via annotation or API. There is no UI path for per-workload HPA revert, which is a documentation gap worth knowing before you rely on this in an incident.

Rebalancing plan preview. Before a rebalancing operation executes, Cast AI generates a reviewable plan. You can inspect which nodes the system will replace, which workloads it will migrate, the projected cost impact, and the expected savings. The plan uses the same algorithms as the live autoscaler, so it reflects the actual cluster state rather than a theoretical model. This preview step helps you catch configuration issues or unexpected scope before the system disrupts anything.

In-place resize and 1-by-1 pod restart. From Workload Autoscaler v0.53.0, in-place pod resizing is available for supported workloads. This applies resource changes without restarting the pod, which eliminates a class of disruption entirely. From v0.57.0, the 1-by-1 pod restart strategy applies vertical adjustments by rolling pods one at a time rather than all at once, limiting the blast radius if a resized pod fails to come back healthy.

The rightsizing safety data cited above in Key Takeaways makes the failure mode picture concrete. Bidirectional rightsizing raises starved workloads as well as cuts over-provisioned ones, which is why OOMKills fell while costs dropped simultaneously. One-directional cost cutting would have produced the opposite result on OOMKills. The bidirectionality is the mechanism that makes the outcome safe, not a side effect of it.

Audit Log and Observability

Every automated action produces a permanent record. The audit log is the primary accountability mechanism.

Audit log structure. Each audit log entry contains a timestamp, the operation name, and the initiator, which is either a user email address or the name of the policy that triggered the action. You can expand entries to view full YAML or JSON details and filter them by rebalance ID, node ID, node status, applied policy, node template version, or config version. This means you can reconstruct the full decision context for any historical action: what policy version was running, what node template was active, which rebalancing plan triggered the drain, and who or what initiated it.

The ListAuditEntries API makes the audit log programmable. A basic query against your cluster’s audit history looks like this:

# Retrieve recent audit log entries for a cluster
# Full API reference: https://docs.cast.ai/reference
curl -X GET \
  "https://api.cast.ai/v1/audit/log?clusterId=YOUR_CLUSTER_ID&limit=100" \
  -H "X-API-Key: YOUR_CAST_AI_API_KEY"

You can pull the full event history into your own data pipeline, compare against incident timelines, or drive custom alerting on specific operation types by filtering the response by operationType or policyName.

Prometheus metrics. All Cast AI operational metrics are Prometheus-scrapable. They cover node type distribution, Spot versus On-Demand counts, costs, utilization, and optimization state. Grafana dashboards are supported. Compute hourly cost metrics can be projected to monthly estimates using the standard multiplier. If your observability stack already ingests Prometheus metrics, Cast AI integrates without additional tooling.

Cost attribution. Real-time cost visibility is available by cluster, namespace, and workload. Allocation Groups let you slice costs by any label combination without requiring custom tooling or post-processing. This matters for teams that need to show the business impact of optimization work, or attribute infrastructure costs to specific product teams.

Notifications. PagerDuty webhooks and Slack integration are available for real-time cluster event alerts. This includes drain failures, rebalancing completions, Spot interruption events, and optimization state changes. If you want to be paged when a drain fails due to a PDB violation, that is a configuration option. If you want a Slack message when a rebalancing run completes, that is also configurable. The notification layer puts Cast AI events into whatever incident management workflow your team already uses.

The Path to Trusting Automation

The onboarding path for Cast AI automation is explicitly incremental. The platform is designed for teams that want to build confidence before delegating full control.

Recommend-only mode. This is the documented starting point. Cast AI generates recommendations for workload rightsizing and node optimization but does not apply them. You review the recommendations in the console, compare them against your own analysis or historical data, and validate that the system is reading your cluster correctly. This phase has no risk. The recommendations are advisory. Teams typically spend one to two weeks in this mode before enabling automation for non-critical workloads.

Scoped automation via whitelisting mode. After validating recommend-only output, most teams enable automation for a controlled subset of workloads first. Whitelisting mode (v0.58.3+) makes this explicit by requiring opt-in per workload. You annotate the workloads you want automated, verify behavior across a few deployment cycles, and expand the scope when you are satisfied with the results.

Full autonomous operation. Once a team has seen automation operating correctly across a meaningful portion of their workload footprint, expansion to full autonomous mode is straightforward. At this point, the audit log and Prometheus metrics provide the continuous visibility needed to catch any anomalies quickly. The exclude annotation remains available for any workload that should stay outside automated control permanently.

In practice, full autonomous operation means the audit log surfaces every action Cast AI takes — node replacements, pod rightsizing, rebalancing runs — with timestamps and policy attribution, without any operator input required for routine changes. Grafana dashboards show cost trends and utilization curves shifting in the right direction over days and weeks. Slack or PagerDuty alerts fire for edge cases that warrant attention, such as a PDB-blocked drain or a Spot interruption on a constrained node pool, while the cluster handles everything else without manual involvement.

Guardrails vs. Karpenter and Native VPA

The open-source tooling in this space is strong but leaves specific gaps that matter at production scale. It is worth being precise about what those gaps are.

Karpenter. Karpenter provisions nodes in 45 to 60 seconds, which is significantly faster than the 3 to 4 minutes typical of the Cluster Autoscaler. It is an excellent node provisioner. What it does not do: workload-level rightsizing, HPA conflict resolution, Spot interruption prediction with multi-hour horizons, savings threshold pre-flight for rebalancing operations, cost attribution by namespace or workload, or audit logging of the kind described above. Karpenter nodes can be provisioned and terminated, but there is no cost-aware rebalancing with a configurable abort threshold. There is no PDB revert annotation on failed drains. Cast AI augments Karpenter by adding the workload optimization and governance layer on top of its node provisioning speed.

Native VPA. Vertical Pod Autoscaler in Off mode gives you recommendations without applying changes, which is safe but requires manual execution. In Auto mode, it applies changes via pod restart, but it cannot run alongside HPA when both scale on the same resource metric. This VPA/HPA conflict is a documented limitation of native Kubernetes tooling. The workaround is to use VPA on CPU and HPA on a custom metric, which requires either a custom metrics setup or accepting that one of your scaling mechanisms is blind to its counterpart. Cast AI’s Workload Autoscaler resolves this architecturally: HPA handles traffic spikes, vertical rightsizing handles baseline efficiency, and the HPA converter ensures the two remain in sync when requests change.

The gap this creates. Teams running Karpenter and native VPA can get strong node provisioning speed and workload-level recommendations, but closing the loop from recommendation to automated action, with guardrails enforced at execution time, requires either significant custom tooling or an autonomous platform. The audit trail, savings threshold pre-flight, PDB revert logic, HPA conflict resolution, and Spot interruption prediction that Cast AI provides do not have direct equivalents in open-source Kubernetes tooling today. That is not a knock on the open-source tools. They were designed for different scopes. It is a statement about what teams need to – build or buy – if they want to operate autonomously at scale with full accountability. For teams evaluating where autonomous optimization fits in their Kubernetes cost optimization strategy, the guardrail and audit controls described above are where the conversation usually starts. Get a quote to understand what running Cast AI at scale costs.

If you want to evaluate Cast AI’s recommendations against your own cluster before enabling any automation, the recommend-only mode gives you that view immediately.

Frequently Asked Questions