An agentic runbook is an AI-powered automation that observes Kubernetes cluster state continuously, selects the appropriate remediation without human input, and executes multi-step recovery workflows end to end. Unlike static scripts or traditional automated runbooks, agentic runbooks make decisions: they detect anomalies, reason about context, and verify that fixes actually worked. The result is a closed-loop system where known failure patterns resolve autonomously, without a human in the loop.
2:17 AM. Your on-call engineer gets paged. A pod in production just OOM-killed for the third time this week. The runbook says: check memory limits, calculate a safe ceiling, update the deployment manifest, apply, and verify. Ten minutes of work. Maybe fifteen if the engineer is half-asleep.
By the time they finish, it has happened again.
The problem is not the runbook. The runbook is correct. The problem is the gap between the alert firing and a human bridging it. Kubernetes does not wait. Pods restart, traffic spikes, Spot nodes disappear. The cluster operates at machine speed. Your remediation does not.
This is the alert-to-action gap. It is where most operational overhead lives, and it compounds at scale.
Three Levels of Runbook Maturity
Level 1: The Runbook
A runbook is a standardized procedure document. It captures expert knowledge: what to check, in what order, and what decisions to make. Runbooks encode tribal knowledge, reduce cognitive load during incidents, and onboard engineers faster.
But a runbook is a document. It requires a human to read it, execute it, and verify the outcome.
At 2 AM, that is expensive. At scale, it is untenable. Kubernetes workloads generate dozens of events daily: pod restarts, scaling events, and node replacements. Manual runbook execution at that frequency is not sustainable.
Level 2: The Automated Runbook
Automated runbooks encode procedures as executable workflows. Instead of reading steps and clicking buttons, an engineer builds a script or pipeline that executes them. Triggered by events, schedules, or on demand, automated runbooks reduce MTTR and eliminate human error on known, repeatable procedures.
This is real progress. Automated runbooks handle the predictable. They run faster than humans and do not require someone to be paged at 2 AM for a scenario the team has already solved.
The limitation: they still require someone to define the trigger, scope the fix, and verify the result. Automated runbooks execute scripts. They do not make decisions.
Level 3: The Agentic Runbook
An agentic runbook is an AI agent that continuously observes cluster state, reasons about which remediation applies, executes multi-step workflows autonomously, validates the resolution, and escalates to humans only when encountering something genuinely novel.
The distinction matters. Automated runbooks execute. Agentic runbooks decide.
An agentic runbook does not wait for an alert. It detects the anomaly, selects the appropriate response, applies the fix, and confirms the outcome. The alert-to-action gap collapses to near zero. The engineer wakes up to a resolved incident, not a page.
| Manual | Automated | Agentic | |
|---|---|---|---|
| Decision-making | Human | Pre-scripted logic | AI agent (context-aware) |
| Trigger | Human-initiated | Event or schedule | Continuous observation |
| Response time | Minutes to hours | Seconds to minutes | Near-zero (sub-second detection) |
| Human involvement | Required every time | Required to define and verify | Escalation only |
| Scalability | Linear with headcount | Limited by script maintenance | Scales with cluster, not team size |
Three Kubernetes Agentic Runbook Scenarios
Scenario 1: OOM Event Handling
A pod OOM-kills when its memory consumption exceeds the configured limit. The typical manual response: an alert fires, an engineer checks metrics, recalculates the limit, updates the manifest, applies the change, and watches the pod recover. Ten to fifteen minutes. Repeated every time it happens.
An agentic runbook handles this differently:
- Detects the OOM kill event in real time
- Analyzes historical memory consumption patterns for the affected workload
- Computes a corrected memory limit, accounting for usage variance and safe headroom
- Applies the corrected memory limits. On K8s 1.27+ clusters with in-place resize enabled, this happens without a pod restart. On older clusters, Cast AI applies on the next scheduling event.
- Monitors the pod post-change and confirms recovery
- Logs the full decision trail for audit
The engineer is not in this loop. The resolution happens before they would have finished reading the alert.
At scale, this matters. A cluster running 500 microservices will see OOM events regularly. Handling each one manually is not viable. Handling them autonomously, consistently, and with full observability is operational leverage.
Scenario 2: Spot Interruption Recovery
Spot instances save significant money. They also disappear.
Cloud providers give anywhere from 30 seconds to 2 minutes of warning before reclaiming a Spot node. A human cannot reliably respond in that window. An agentic runbook can.
The sequence runs automatically:
- Detects the interruption signal from the cloud provider
- Provisions an on-demand replacement node immediately
- Migrate workloads from the interrupted node to the replacement. For stateful workloads with persistent connections, graceful shutdown is used instead.
- Monitors the Spot market for the instance type
- Returns workloads to Spot when price and availability normalize
- Terminates the on-demand node, restoring cost efficiency
For StatefulSets with EBS-backed PVCs, CSI detach and re-attach adds a brief unavailability window; size your PDB accordingly.
This loop runs without human involvement. Engineers see it in logs after the fact. The cluster barely notices the interruption.
Without an agentic runbook, Spot interruptions mean scrambled traffic, degraded SLOs, and engineers manually rebalancing workloads at inconvenient hours. With one, Spot becomes a reliable cost lever rather than a reliability liability.
Scenario 3: Node Consolidation and Bin-Packing
Kubernetes clusters waste compute. The Cast AI Kubernetes Benchmark Report found that clusters average 10% CPU utilization and 23% memory utilization. That waste is not a scheduling failure. It is a consolidation failure: workloads spread across underloaded nodes that nobody is actively reclaiming.
Fixing this manually means: identify underutilized nodes, confirm safe workload migration, cordon nodes, drain pods, verify new placement, and delete nodes. Repeat for every qualifying node. Nobody does this daily. The waste accumulates.
An agentic runbook runs this as a continuous loop:
- Scans the cluster for underutilized nodes
- Evaluates whether running workloads can be safely rescheduled onto fewer nodes
- Migrates workloads using Container Live Migration, zero downtime for stateless workloads; stateful workloads use graceful shutdown
- Deletes now-empty nodes
- Restarts the loop
The cluster stays consolidated. Node count drops. Cloud bill drops. No human time spent.
This is not a batch job run on a schedule. It is a continuous optimization loop responding to the cluster’s actual state. Workloads that just scaled down free up capacity, the agent finds it, and consolidation happens automatically.
What Agentic Runbooks Actually Require
For agentic runbooks to work on Kubernetes infrastructure, three things must be true.
Observability. The agent needs real-time visibility into pod behavior, node utilization, resource requests versus actual consumption, and cloud provider signals. Without this, it cannot reason about cluster state correctly.
Actuation. Read-only access produces recommendations. Agentic runbooks require write access: the ability to resize workloads, provision nodes, migrate pods, and delete empty capacity. Tools that stop at recommendations are still leaving a human in the loop.
Verification. A runbook that applies a fix and stops is still an automated runbook. An agentic runbook confirms the fix worked. If the pod OOM-kills again after the memory limit increase, the agent escalates rather than looping into an incorrect remediation indefinitely.
Agentic runbooks must also respect the controls you already have in place. Cast AI honors PodDisruptionBudgets during eviction: if a PDB blocks removal of a pod, the agent skips to the next candidate rather than forcing the eviction. Resource request changes propagate correctly alongside HPA, so the autoscaler sees updated requests and adjusts scaling behavior accordingly.
These requirements explain why most observability platforms have not crossed into agentic remediation. Observability is the easier part. Actuation and verification on production infrastructure require both technical capability and organizational trust. PagerDuty, Dynatrace, and New Relic’s SRE Agent all surface recommendations, but they still require human confirmation before touching production systems. The loop is not closed.
Cast AI’s APA Platform: An Agentic Runbook Engine for Kubernetes
Cast AI’s Application Performance Automation (APA) platform implements agentic runbooks for Kubernetes infrastructure. Every core capability is a closed-loop remediation workflow: detect, optimize, verify.
The scenarios above are not hypothetical. They run continuously on Cast AI-managed clusters.
Workload rightsizing operates autonomously. When CPU requests diverge from actual usage, Cast AI recomputes optimal requests and applies them in-place. No deployment restart. No human involvement. The workload gets what it needs, not what someone estimated six months ago when the manifest was written.
Bin-packing runs as a persistent loop via Cast AI’s Evictor. Underutilized nodes are identified, workloads migrated with Container Live Migration for zero-downtime transfers, and nodes deleted. The cluster compacts itself continuously.
Spot management handles the full lifecycle: provisioning replacements on interruption, migrating workloads, monitoring the market, and returning to Spot when conditions allow. The cost savings from Spot persist without engineers babysitting the instance pool.
Based on Cast AI customer data, organizations that enable agentic remediation policies reduce infrastructure incidents requiring human intervention by up to 80%. The gains are largest in clusters with high workload churn and diverse instance types.
Customer outcomes reflect what continuous agentic remediation produces at steady state. Akamai reduced cloud costs by 40-70% (vs. unoptimized baseline). Bud reached 90%+ resource utilization (vs. unoptimized baseline). Wio Bank cut compute costs by up to 70% (vs. list-price On-Demand). Yotpo reduced cloud spend by 40% (vs. unoptimized baseline).
These are not one-time optimization results. They are what happens when the cluster is always being managed, not periodically reviewed.
How to Get Started with Agentic Runbooks on Kubernetes
Cast AI’s Application Performance Automation (APA) platform connects to existing clusters via a single Helm chart. No infrastructure changes are required. Start in read-only mode to see what would change, then enable policies incrementally as confidence grows.
- Install the Cast AI agent:
helm install castai-agent castai-helm/castai-agent --namespace castai-agent --create-namespace - Cast AI connects with read-only access by default. No changes are made until you explicitly enable optimization policies.
- Enable rightsizing for a single deployment first. Observe. Then expand scope.
- Review every proposed change in Deferred mode before enabling autonomous execution.
Platform teams control the scope. You specify which namespaces Cast AI can touch. Changes go into a review queue (Deferred mode) until you enable per-policy autonomous execution. Every action is logged with the reasoning that triggered it. Novel scenarios surface as recommendations rather than autonomous changes.
How to Evaluate Whether Your Cluster Needs Agentic Runbooks
Not every cluster needs agentic automation on day one. But certain signals indicate that manual and automated runbooks have hit their ceiling.
Your cluster is a strong candidate when on-call engineers handle the same failure patterns repeatedly (OOM kills, Spot interruptions, node pressure events), when cluster CPU utilization sits below 30% with no active consolidation running, or when Spot adoption is blocked by reliability concerns rather than workload requirements.
A practical first check: run kubectl top nodes and compare actual utilization against your resource requests. If utilization is below 30% of requested capacity across most nodes, you are paying for idle compute that a continuous consolidation loop would reclaim automatically.
Platform teams spending more than a few hours per week on remediations that follow predictable patterns have already reached the point where agentic automation pays for itself.
The Operational Shift
Kubernetes was designed to be operated by software, not humans. The declarative control plane, the scheduler, the controller loop: all of it exists so the cluster can self-manage. Agentic runbooks extend that principle to the operational layer that Kubernetes itself does not cover: resource efficiency, cost optimization, and incident response.
The CNCF ecosystem now spans 230+ projects, each with its own alert patterns, failure modes, and remediation needs. The surface area alone makes manual runbook execution structurally impossible at scale.
The question for platform and DevOps teams is not whether agentic runbooks are useful. It is whether you want engineers spending hours on remediations a system can handle autonomously, or on work that requires human judgment: architecture decisions, incident post-mortems, capacity strategy, and the novel failure modes that no runbook has seen before.
The alert-to-action gap drains engineering time and compounds operational debt. Closing it is an engineering problem with an engineering solution.
Start optimizing your cluster
CAST AI automates Kubernetes cost, performance, and security management in one platform, achieving over 60% cost savings for its users.
Frequently Asked Questions
An automated runbook is a script or workflow that executes predefined steps when triggered by an event or schedule. It runs faster than a human but cannot make decisions: if the situation is ambiguous or the fix does not work, it stops or repeats the same incorrect action. An agentic runbook adds a reasoning layer: it evaluates cluster state, selects from multiple possible remediations, applies the fix, verifies the outcome, and escalates only when the situation falls outside recognized patterns.
When a cloud provider signals a Spot instance interruption (typically 30 seconds to 2 minutes of warning), an agentic runbook immediately provisions an on-demand replacement node, migrates running workloads, and monitors Spot pricing to return workloads to Spot once conditions normalize. The entire sequence runs without human involvement. For StatefulSets with EBS-backed PVCs, CSI detach and re-attach adds a brief unavailability window; Pod Disruption Budgets should be sized to absorb this transition.
The fastest path is installing an agent in read-only (observation) mode, so you can see what it would change before enabling autonomous execution. Cast AI’s Application Performance Automation (APA) platform installs via a single Helm chart and begins surfacing optimization recommendations within minutes. Enable policies incrementally: start with workload rightsizing for a single namespace, review proposed changes in Deferred mode, then expand scope as confidence grows.
