OOMKilled and Exit Code 137: Why Kubernetes Kills Your Pods and How to Stop It

What does OOMKilled (exit code 137) mean?

OOMKilled is a Linux kernel feature, not a Kubernetes feature. The cgroup memory controller enforces limits at the kernel level. Kubernetes reads the container exit code and sets reason: OOMKilled in pod status — it’s a reporter, not the enforcer.

Why Kubernetes kills a pod with exit code 137

Three distinct scenarios produce OOMKilled. Conflating them leads to the wrong fix.

Scenario 1: Container memory limit reached

The most common case. When a container’s memory consumption crosses its cgroup limit, the kernel kills that container only — the rest of the pod keeps running. Kubelet restarts the container per the pod’s restartPolicy.

On Kubernetes 1.28+ with cgroup v2, memory.oom.group=1 kills all processes in the container together rather than just the offending process. This is the correct behavior for multi-threaded applications; previously, a background thread crossing the limit could leave the main process in a broken state.

Init container OOMKills: If an init container gets OOMKilled, the pod never transitions to Running — it stays in Init or Init:OOMKilled state and restarts. This is distinct from a running container OOMKill, where the pod is already running and the container restarts within it. Check kubectl describe pod for init container terminated state.

Sidecars as hidden culprits: Istio proxies, Datadog agents, and other sidecars run in the same pod and draw from the same node memory. An app container correctly sized at 256Mi can still push the pod toward OOM if an Istio sidecar is consuming another 100–150Mi that was never accounted for in the limit. Always inspect every container in the pod, not just the application container.

QoS class determines which containers the kernel targets first under node memory pressure:

Scenario 2: Node memory pressure and kubelet eviction

Kubelet monitors node-level memory via eviction thresholds (default: memory.available < 100Mi). When the node approaches exhaustion, kubelet proactively evicts pods — BestEffort first, then Burstable. This is distinct from an OOMKill:

Scenario 3: Node-level kernel OOM (container under its own limit)

If kubelet eviction doesn’t shed pods fast enough, the kernel OOM killer fires at the node level. A container can be OOMKilled here even if it never exceeded its own cgroup limit — it was simply the lowest-scored candidate in the kernel’s OOM priority calculation at that moment.

This appears as OOMKilled in pod status, identical to Scenario 1. Distinguish via the node syslog:

grep "Out of memory: Killed process" /var/log/syslog

If the output does not contain the string memcg, this is a node-level OOM — not a container cgroup limit breach. The fix is different: ensure sufficient node memory headroom using kube-reserved memory configuration, not by changing container limits that weren’t actually exceeded.

How to diagnose OOMKilled (kubectl describe, events, logs)

Start with kubectl describe. Look for the terminated state in the container status block:

kubectl describe pod <pod-name> -n <namespace>

You’re looking for:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Pull logs from the previous (killed) container instance:

kubectl logs --previous <pod-name> -n <namespace>

Check recent events:

kubectl get events --sort-by=.lastTimestamp -n <namespace>

For cluster-wide visibility in Prometheus:

kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}

To understand the memory trajectory that led to the kill, query actual working set memory. Use container_memory_working_set_bytes — not container_memory_usage_bytes, which includes page cache the kernel will reclaim before triggering OOM:

quantile_over_time(0.95, container_memory_working_set_bytes{container!=""}[7d:5m])

This p95 value over 7 days is your baseline for setting memory requests. If your current limit sits below this number, OOMKilled will keep happening.

For PSI (Pressure Stall Information) — a leading indicator of memory starvation before the OOM killer fires:

node_pressure_memory_stalled_seconds_total

Once you know the right values, apply them immediately:

kubectl set resources deployment/my-app -c my-container --requests=memory=256Mi --limits=memory=512Mi

How to fix OOMKilled

Set memory requests and limits correctly

• Memory request: set at p95 of container_memory_working_set_bytes over 7 days
• Memory limit: set at p99 or higher — 2x the request is a reasonable starting point without tail data
• Guaranteed QoS (requests == limits, non-zero): grants oom_score_adj=-997 — the lowest kill priority, last to be targeted under node pressure. Use this for latency-sensitive production workloads where any restart is unacceptable. The scheduler treats the full request as reserved capacity, so cost is higher.
• Burstable QoS (requests < limits): the right choice for most workloads. Set requests at p95 actual usage, limits at p99+ or 1.5–2x requests. Pods can absorb traffic spikes without exhausting node capacity.

If you’re using VPA for recommendations without enforcement:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"  # recommendations only, no enforcement

Check .status.recommendation.containerRecommendations for the suggested values. Valid updateMode values: Off, Initial, Recreate, Auto.

Find and fix memory leaks

If working set grows unbounded between restarts, you have a leak — raising the limit only extends the time between kills. JVM applications deserve specific attention: setting -Xmx equal to the container limit ignores off-heap allocation (metaspace, code cache, threads). Rule of thumb: -Xmx = container_limit × 0.75. The remaining 25% covers non-heap JVM overhead.

Why adding more memory is not the fix

Blanket limit increases across your fleet don’t prevent OOM kills — they raise the average everywhere while the specific containers that actually need more memory keep dying. According to the Cast AI 2026 Kubernetes Optimization Report, average memory utilization across production clusters is 20% with 79% overprovisioning. Clusters with generous memory padding were still averaging 40–50 OOM kills per monitoring interval. The memory was provisioned — it was provisioned to the wrong workloads.

How to prevent OOM kills at scale (automated rightsizing)

Manual sizing works for a handful of services. At 50+ deployments with variable traffic, it breaks down. Memory profiles change with code releases, seasonal load, and dataset growth. A limit that was correct three months ago may be wrong today.

Cast AI Workload Optimization tracks container_memory_working_set_bytes continuously, identifies containers approaching their limits, and raises those limits proactively — before an OOM kill, not after. For containers that are significantly overprovisioned, it brings limits down, recovering headroom for workloads that actually need it.

On Kubernetes 1.33+, In-Place Pod Resizing adjusts memory limits without restarting pods. For stateful workloads, this matters: a traditional VPA-style resize requires a pod restart that may itself cause a brief outage. In-place resizing eliminates that constraint.

Cast AI uses PSI (Pressure Stall Information) metrics as leading indicators. PSI measures time processes spend stalled waiting for memory — it signals starvation before the OOM killer fires. Acting on PSI lets the system raise a limit before the container dies.

After enabling automated rightsizing, OOM kills dropped to near zero across clusters that were previously averaging 40–50 kills per interval (Cast AI 2026 Kubernetes Optimization Report). The rightsizing engine analyzes all resource dimensions simultaneously — when it corrects memory limits, it also surfaces CPU overprovisioning in the same pass. Provisioned CPUs dropped by roughly half in the same clusters. Memory and CPU overprovisioning tend to coexist; fixing one without the other leaves the job half done.

For cluster-by-cluster breakdowns of where overprovisioning concentrates and which workload types are most OOM-prone, see the Cast AI 2026 Kubernetes Optimization Report.

FAQ