, ,

Disk Full or IOPS Throttled? The Two Storage Emergencies That Look Identical at 2 a.m.

A Kubernetes storage runbook for on-call engineers: identify exhausted capacity, inode pressure, and performance limits, then choose the right fix.

Laurent Gil Avatar
disk-full-vs-iops-castai

A full disk runs out of writable space or inodes; an IOPS bottleneck limits how many storage operations can complete each second. Check the affected mount with df -h and df -i, then correlate node-level iostat with cloud volume metrics. Capacity exhaustion calls for safe cleanup or expansion; confirmed throttling calls for performance tuning.

Both conditions can cause slow writes, timeouts, or failed requests, but symptoms alone do not identify the cause. This runbook separates filesystem capacity, device-level latency, and provider limits. It uses Kubernetes with AWS EBS examples; commands, permissions, and available metrics depend on your cluster and storage driver. Capacity and performance problems can also occur together.

One caveat decides whether you finish the incident or chase a ghost, since each check runs in a different place. df reads the pod’s mount namespace, iostat reads the hosting node, and cloud metrics come from the control plane. Bridge them with one chain: Pod, then Node, then PersistentVolume, then cloud volume ID, then block device name. Skip a link and you read a healthy pod’s stats while the node’s device is saturated.

How do you tell a full disk from an IOPS bottleneck?

  • Full disks and IOPS throttling can both disrupt writes. Confirm the cause with capacity, latency, and provider-limit measurements.
  • Check 1 is space: run df -h inside the container and df -i for inodes, then read the node’s DiskPressure condition. Even with free space, zero inodes fails exactly like a full disk.
  • Check 2 is device behavior: compare iostat latency and queue depth with a healthy baseline. High %util alone does not prove saturation on parallel devices.
  • Check 3 is the provider ceiling: correlate demand with volume and instance IOPS or throughput limits. Use supported exceeded-check metrics and, for gp2, BurstBalance.
  • Choose the remedy for the confirmed constraint. Raising provisioned IOPS does not free space; increasing gp3 capacity does not automatically raise its configured IOPS. Some other disk types tie performance to size.

Why a full disk and an IOPS bottleneck look identical

A failed write can produce an application error, timeout, or process exit. Kubernetes restarts a container according to its restart policy when it exits; a slow disk does not automatically cause a restart. Inspect application logs and pod events alongside storage metrics. Multi-Attach errors are attachment conflicts and require a separate investigation.

A busy device may queue requests, increasing write latency. A filesystem with little free space can also have allocation problems, but the effect depends on the filesystem and workload. Neither a latency spike nor a crash loop proves that a disk is full. Check free bytes, inodes, and the actual write error before choosing a remedy.

The observability layer hides the distinction further. Dashboards track request duration, error rate, and queue depth, and none of them know why the write took 800ms. Container metrics show CPU and memory, while the pod is neither CPU-bound nor OOM-killed. So the dimension that would settle it sits in a namespace most dashboards never query.

Alerting completes the confusion. A PVC usage alert fires only if someone configured it, disk pressure fires on the node rather than the pod, and IOPS throttling often alerts only once the SLO burns. The engineer paged at 2 a.m. opens a crash-looping StatefulSet, sees a PVC at 61 percent, and looks elsewhere. Or expands a volume at 94 percent and watches the pod crash anyway, because capacity was never the constraint.

Check 1: is it space?

Because the space check is cheapest and rules out the most common cause, it runs first.

kubectl exec -it payments-db-0 -n payments -- df -h /var/lib/postgresql/data

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    196G  186G  2.1G  99% /var/lib/postgresql/data

Commands and outputs are illustrative, not measurements from a production incident. Replace example names, volume IDs, paths, and timestamps with your own. Run read-only checks first; review permissions and backups before making storage changes.

When the container has no shell or no df

Distroless images ship neither a shell nor coreutils, so that first command fails with executable file not found instead of a disk report. Such a failure does not mean the pod is fine. Read the node instead, where the kubelet keeps its view of the filesystem.

# Read the affected pod and PVC statistics; requires node proxy access and jq.
kubectl get --raw "/api/v1/nodes/ip-10-0-14-22.ec2.internal/proxy/stats/summary" \
  | jq '.pods[]
    | select(.podRef.namespace == "payments" and .podRef.name == "payments-db-0")
    | .volume[]?
    | select(.pvcRef.name == "postgres-data-payments-db-0")
    | {name, pvcRef, availableBytes, capacityBytes, usedBytes, inodesFree, inodes}'

The kubelet statistics API uses fields such as usedBytes and capacityBytes. A missing result can mean the driver does not report these statistics, the identifiers do not match, or access is denied; it is not proof of free space. If necessary, have an authorized operator inspect the actual PVC mount from the node. Measuring the node root filesystem alone does not measure the PVC.

Inodes, then node pressure

A volume at 99 percent is a space problem. If the number is comfortable, run the inode check, since it produces an identical failure with a different number on the Use% column.

kubectl exec -it payments-db-0 -n payments -- df -ih /var/lib/postgresql/data

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/nvme1n1    12.8M  12.8M     0  100% /var/lib/postgresql/data

Zero free inodes means the filesystem cannot allocate a new file despite gigabytes of free space, so every write that creates a file fails with No space left on device. Mail queues, session stores, and training jobs writing a checkpoint per step hit this long before they exhaust blocks.

Next, the node. Kubernetes tracks disk pressure node-wide and evicts pods before the volume fills.

kubectl describe node ip-10-0-14-22.ec2.internal | sed -n '/Conditions:/,/Addresses:/p'
kubectl get events --field-selector reason=Evicted --sort-by=.lastTimestamp -A
# DiskPressure   True  ... KubeletHasDiskPressure  kubelet has disk pressure

DiskPressure: True indicates a node filesystem space or inode threshold has been crossed. Default hard thresholds include nodefs.available<10% and imagefs.available<15%, but cluster configuration can override them. The kubelet attempts reclamation before evicting pods; eviction ranking considers resource usage relative to requests and pod priority. A full PVC is a separate condition and does not necessarily trigger node disk pressure. See Kubernetes node-pressure eviction.

Check 2: is the device saturated?

If space is fine, measure whether the device keeps up with demand. Start from the pod, resolve its node with kubectl get pod -o wide, find the bound claim, then take the volume handle from the PV’s CSI attributes.

kubectl get pod payments-db-0 -n payments -o wide
kubectl get pvc postgres-data-payments-db-0 -n payments -o jsonpath='{.spec.volumeName}'
kubectl get pv pvc-8f2c1e40 -o jsonpath='{.spec.csi.volumeHandle}'
# returns vol-0a1b2c3d4e5f60718; on the node, lsblk -o NAME,SERIAL maps it to /dev/nvme1n1

Now run iostat on that node, against that device. The -z flag omits idle devices, so anything missing is doing nothing, not missing data. Use iostat -xz 5 3 and act on the last interval, because the first still carries counters from before the command started.

# Requires sysstat; run on the node, not in the pod. Output columns vary by version.
# -z hides idle devices; read the LAST of three 5-second intervals
sudo iostat -xz 5 3 /dev/nvme1n1

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme1n1        312.40  24876.0     0.00   0.00    1.82    79.6  1890.20 60486.0    14.20   0.75   14.61    32.0    0.00     0.0     0.00   0.00    0.00     0.0   11.30   12.40   27.94   99.7

The pod to device mapping matters more than the flags, since a shared node hosts many volumes.

Reading utilization and queue depth

%util is the percentage of time during which I/O requests were issued to the device. aqu-sz, called avgqu-sz in older sysstat releases, describes the average number of outstanding requests. Compare both with a healthy workload baseline; queue depth has no universal saturation threshold.

On NVMe, RAID, and other devices that process requests in parallel, high %util does not establish that maximum performance has been reached. Evaluate queue depth, request size, IOPS, throughput, and await together. A high await indicates longer request completion time, including queueing, but does not identify the underlying cause by itself.

The remaining columns help characterize the I/O pattern. They support the diagnosis; they do not replace the filesystem and cloud-limit checks.

What the wait and merge columns say

w_await is the average time for write requests to complete, including time in the queue and time being serviced. Interpret it against the workload baseline, I/O size, and storage service expectations. There is no universal 10 ms threshold that distinguishes a faulty device from contention. Compare the same time interval across application latency, device statistics, and provider metrics.

%wrqm describes write requests merged before reaching the device. A low value alone cannot identify a database workload or prove random I/O; the scheduler and I/O path affect merging. Device statistics also do not identify which pod generated the traffic.

# per-cgroup I/O on cgroup v2; read_total is rbytes, write_total is wbytes
for cg in /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/*/; do
  echo "$cg"; cat "$cg/io.stat"
done

This cgroup v2 example covers only Burstable pods on a systemd-style hierarchy. Guaranteed and BestEffort pods use other paths, and layouts vary by runtime. io.stat counters are cumulative: take two samples and divide the difference by elapsed time to estimate rates. Match the cgroup to the pod UID and device major/minor number. On cgroup v1, inspect the applicable blkio counters instead.

Check 3: are you hitting the provisioned ceiling?

When capacity is available but latency rises, test whether workload demand reaches a volume or instance performance limit. Do not assume the hardware is healthy or that provisioning is the cause before checking provider health and limit metrics. AWS exposes these signals through CloudWatch.

# AWS CLI v2.15
aws cloudwatch get-metric-statistics \
  --namespace AWS/EBS \
  --metric-name VolumeQueueLength \
  --dimensions Name=VolumeId,Value=vol-0a1b2c3d4e5f60718 \
  --start-time 2026-11-16T01:00:00Z \
  --end-time 2026-11-16T02:15:00Z \
  --period 60 \
  --statistics Average Maximum

VolumeQueueLength alone does not prove throttling. On supported Nitro configurations, use VolumeIOPSExceededCheck and VolumeThroughputExceededCheck. Compare demand with provisioned limits using VolumeAvgIOPS and VolumeAvgThroughput, where available, or derive IOPS from the sum of VolumeReadOps and VolumeWriteOps divided by the period in seconds. Check instance-level EBS limits too. Metric availability varies by platform and attachment type; consult the AWS EBS CloudWatch metric reference.

For Google Cloud or Azure, use the metrics documented for the exact disk type and VM configuration. Compare disk and VM performance limits, observed IOPS and throughput, and latency over the same interval. AWS metric names and eligibility rules do not transfer to those providers.

The AWS EBS volume types guide explains which limits depend on size, provisioned settings, or credits. gp2 deserves an additional check because credit exhaustion can change performance without a configuration change.

Burst credit exhaustion on gp2

gp2 baseline performance is 3 IOPS per GiB, with a minimum of 100 and maximum of 16,000 IOPS. Volumes below 1,000 GiB can burst to 3,000 IOPS while credits remain. Check BurstBalance alongside demand.

For example, a 100 GiB gp2 volume has a 300 IOPS baseline. With a full 5.4-million-credit balance and sustained 3,000 IOPS, the credit budget lasts about 33 minutes: 5,400,000 divided by (3,000 minus 300). A partially depleted balance runs out sooner. Performance returns to baseline after exhaustion; the latency impact depends on demand.

gp3 has no burst-credit model. Its included baseline is 3,000 IOPS and 125 MiB/s. Regional maximums are 80,000 IOPS and 2,000 MiB/s, subject to volume-size and IOPS-to-throughput constraints; Outposts has lower limits. Capacity and provisioned performance can be configured separately. Check the current gp2 and gp3 limits before changing a volume.

The fix for each case

Choose the change that addresses the measured constraint. More than one problem may coexist, and the relationship between capacity and performance depends on the disk type. Re-run the checks after each change to confirm recovery.

ConstraintCheckInterpretationResponse
Filesystem capacity or inodesdf -h and df -i on the affected mountLittle usable space or no free inodes; confirm write errorsSafely remove disposable data or expand a supported PVC. Inode behavior on expansion depends on the filesystem.
Node disk pressurekubectl describe node and pod eventsNode filesystem or inode eviction threshold crossedReclaim node storage or expand the appropriate node filesystem; do not assume a PVC resize fixes it.
Device latency or queueingiostat -xz 5 3, mapped to the correct deviceLatency or outstanding requests above the healthy baselineIdentify workload demand and provider limits before changing capacity or performance.
Volume or instance performance ceilingSupported EBS exceeded-check metrics, IOPS, throughput, and gp2 BurstBalanceDemand reaches a documented limit or credits are exhaustedTune supported volume performance, reduce demand, or address the instance ceiling. Evaluate gp2-to-gp3 migration where appropriate.

Space fixes: expansion and cleanup

Kubernetes grows a bound PVC only if the StorageClass sets allowVolumeExpansion: true, which you check before promising a fix. You can raise a PVC storage request on such a StorageClass, and you cannot shrink it afterward, so size the step deliberately rather than doubling on impulse.

kubectl get storageclass gp3 -o jsonpath='{.allowVolumeExpansion}'
# true

kubectl patch pvc postgres-data-payments-db-0 -n payments \
  -p '{"spec":{"resources":{"requests":{"storage":"400Gi"}}}}'

kubectl get pvc postgres-data-payments-db-0 -n payments -w

Verify that the CSI driver and filesystem support expansion, then watch PVC conditions and confirm the new size inside the mounted filesystem. If data is disposable, remove only files whose retention and application requirements you understand. Do not delete database files or WAL manually. VACUUM FULL is not an emergency space-freeing shortcut: it rewrites tables, requires additional free space, and takes an exclusive lock. Plan database maintenance separately; see the PostgreSQL VACUUM documentation.

Throughput fixes: IOPS and volume-type changes

EBS Elastic Volumes can change supported volumes without detaching them on compatible instances. gp3 supports independently provisioned IOPS and throughput; io1 and io2 support provisioned IOPS, while st1 and sc1 performance is size-dependent. Do not apply gp3 parameters to every disk type. Confirm supported options in the modify-volume reference.

Once the metrics confirm a performance bottleneck, follow the EBS gp3 and GKE Hyperdisk tuning guide to choose settings, account for provider modification limits, and validate the application outcome.

aws ec2 modify-volume \
  --volume-id vol-0a1b2c3d4e5f60718 \
  --volume-type gp3 \
  --iops 8000 \
  --throughput 500

aws ec2 describe-volumes-modifications \
  --volume-ids vol-0a1b2c3d4e5f60718

The state moves from modifying to optimizing to completed. During optimization, performance transitions between the source and target configurations. Completion may take hours and can exceed 24 hours; six hours is not a maximum. AWS permits up to four modifications per rolling 24 hours, and the previous modification must be completed before the next. Check modification progress rather than assuming a fixed wait.

Why neither fix covers the other’s problem

Expanding a 200 GiB gp3 volume to 400 GiB adds capacity and nothing else. Provisioned IOPS stays fixed, so a volume throttled at 3,000 IOPS remains throttled. On gp2 the expansion raises baseline IOPS at 3 per GiB, but it raises price linearly and leaves the burst mechanic in place. That is an expensive way to buy throughput.

Raising IOPS on a full volume changes no free space. Writes still fail with No space left on device, now with performance you paid for and cannot use. The distinction sounds obvious written down, yet it is easy to lose at 2 a.m. when the alert says “disk” and the dashboard says “slow.”

When it’s neither: network storage latency

If capacity and provider limits do not explain the incident, widen the investigation. Application locks, filesystem behavior, remote filesystem RPCs, attachment failures, and instance limits affect different parts of the I/O path. Compare measurements from the same operation and time window before attributing the delay to the network.

Availability Zone mismatch: an attachment failure

An EBS volume and its EC2 instance must be in the same Availability Zone. A mismatch can leave a pod unschedulable or unable to attach its volume; it does not create a slow cross-AZ EBS connection. Check pod events, PV node affinity, and the node zone label. Correct placement or plan a snapshot-based migration when a different zone is required. See EBS attachment requirements.

Client-side network filesystem latency

For NFS and EFS, local block-device statistics do not directly describe remote filesystem RPC latency. Client caching, server load, request size, and the network can all matter. Inspect nfsstat -c for RPC retransmissions and mountstats for per-operation timings, then correlate with server or service metrics. A slow metadata operation is a clue, not proof that a cache setting is wrong.

Instance-level EBS bandwidth and IOPS limits

An instance can reach its EBS bandwidth or IOPS ceiling even when individual volumes have spare provisioned performance. Check the instance type’s EBS-specific limits, not just general network bandwidth. On supported instances, InstanceEBSIOPSExceededCheck and InstanceEBSThroughputExceededCheck help identify this constraint. Increasing volume IOPS cannot remove an instance-level ceiling.

Telling them apart in practice

If application write latency is much higher than device latency, inspect the timing boundaries first. Application measurements may include locks, buffering, transactions, replication, and synchronization. Remote filesystem operations follow a different path. The difference narrows the investigation but does not, by itself, prove a network problem.

Match the response to the evidence: fix zone placement for an attachment failure, investigate RPC and service behavior for remote filesystem delays, or select an instance with sufficient EBS performance for a confirmed instance limit. Changing storage architecture requires an application and data-migration plan.

Preventing the 2 a.m. page

Alert on capacity and performance separately. For an example capacity alert, compare kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes with a threshold chosen for your growth rate and response time; 85% for 30 minutes is only a starting point, not a universal policy. Monitor inode availability where the CSI driver exposes it, and add a faster warning for rapid growth.

For throttling, combine supported provider exceeded-check metrics with workload latency, demand, and queue depth over matching intervals. Monitor gp2 BurstBalance before it reaches zero. Set warning duration and escalation according to the workload SLO; do not use a generic utilization percentage as proof of a provider cap.

How Cast AI addresses this

The gap is operational. Every number above is available to any engineer with kubectl and cloud CLI access, yet the 2 a.m. page still happens. Someone still has to notice a threshold crossing, decide whether it is capacity or performance, and resize while four other alerts fire. The checks tell you what to do, and nothing in a stock cluster does it for you.

Cast AI Storage Autoscaler automates capacity expansion and performance rightsizing for supported Kubernetes storage on EKS and GKE. Policies determine which volumes are managed and which actions are allowed. Capacity expansion and performance tuning address different constraints; supported operations depend on the cloud and disk type.

Start with dry-run mode and inspect proposed actions before enabling changes. Configure thresholds, growth steps, limits, and cooldowns for your workload rather than treating example settings as universal defaults. The Storage Autoscaling documentation describes current support and identifies the feature as early access. It does not replace diagnosis of attachment failures or application-level latency.

For a walkthrough of policies and rollout, read how Kubernetes storage autoscaling works. Start a free trial in read-only mode to explore Cast AI on your clusters before enabling automated changes.

FAQ

How do I know if my disk is full or throttled?

Check free bytes and inodes on the affected mount with df -h and df -i. If capacity is available, compare iostat latency and queue depth with a healthy baseline, then check cloud volume and instance limits. High device utilization alone does not prove throttling. Capacity and performance problems can coexist.

What is Kubernetes disk pressure?

DiskPressure is a node condition triggered by filesystem space or inode eviction thresholds. It is different from a full PVC. The kubelet tries to reclaim resources and may evict pods according to usage relative to requests and priority. Check kubectl describe node and your configured eviction thresholds.

Why is my EBS volume slow but not full?

Possible causes include volume IOPS or throughput limits, instance-level EBS limits, gp2 credit exhaustion, workload contention, and storage health problems. Correlate latency with supported CloudWatch exceeded-check metrics and demand. Increase provisioned performance only when the measurements identify that constraint.

What are gp2 burst credits?

gp2 credits allow a volume to perform above its baseline IOPS. The balance rebuilds when demand is below baseline and drains when demand exceeds it. After exhaustion, performance is limited to baseline until credits recover. Monitor BurstBalance; gp3 does not use this burst-credit model.

Conclusion

Diagnose the affected filesystem first, then the mapped device, then the provider and instance limits. Confirm whether the constraint is capacity, inodes, provisioned performance, or another part of the I/O path. Apply the smallest appropriate change and verify recovery with the same measurements.

Record the pod-to-volume mapping, healthy baselines, escalation thresholds, and rollback or recovery steps in your runbook. Automate recurring capacity and performance adjustments only after the policy and proposed actions have been reviewed.

Cast AI › Blog › Disk Full or IOPS Throttled? The Two Storage Emergencies That Look Identical at 2 a.m.