How to Increase GPU Utilization in Kubernetes (5 Proven Fixes)

Last updated: November 2026

To increase GPU utilization in Kubernetes, first measure what is actually happening with DCGM, separating SM (compute) utilization from memory utilization, because a GPU can read as “busy” while doing almost no useful work. Then apply five fixes, roughly in order of payback: right-size GPU requests, share GPUs with MIG / MPS / time-slicing, bin-pack workloads with GPU-aware scheduling, autoscale GPU nodes to match demand, and queue batch jobs so idle windows get reused.

Most Kubernetes GPU fleets run far below their potential. Average sustained utilization across the production clusters analyzed for Cast AI’s 2026 State of Kubernetes Optimization Report is 5%, measured before optimization. In practical terms, roughly 20 times as much GPU capacity is provisioned as workloads consume at any given moment. The gap is rarely a hardware shortage. It is capacity already paid for and left unconsumed, and the cost of leaving it that way is climbing: for the first time since EC2 launched in 2006, GPU prices are rising, and AWS raised H200 Capacity Block prices 15% in January 2026. The measurement section below shows how to confirm the gap in your own fleet before you change anything.

Key takeaways

• Sustained GPU utilization in the 2026 State of Kubernetes Optimization Report averages 5% before optimization, roughly 20x overprovisioning.
• By provider in the same data: AKS 2%, EKS 5%, GKE 6%. Every managed platform carries the problem.
• GPU utilization is two numbers, not one: SM (compute) activity and memory utilization. A GPU can show high memory use at low compute percentage, and the headline nvidia-smi number hides this.
• Low numbers usually trace to five root causes: oversized requests, one-pod-per-GPU allocation, poor bin-packing, static pools, and no queue for batch jobs.
• Five fixes, in priority order: right-size requests, share GPUs, bin-pack with GPU awareness, autoscale nodes, then queue batch work.
• For inference, GPU sharing is the biggest single lever. An A100 40GB or 80GB partitions into up to 7 MIG instances, though production mixes usually land at 2 to 4.
• At fleet scale this cannot be tuned by hand. Continuous automation for rightsizing, bin-packing, and GPU cost visibility keeps utilization high.

What GPU utilization actually means

In this article, GPU utilization means the percentage of total provisioned GPU compute cycles that produce useful output across a 24-hour period, with AI-Labs clusters excluded. That is the 2026 report definition, and it is harsher than the device-level number most dashboards show.

Two metrics matter underneath it: DCGM_FI_PROF_SM_ACTIVE, the fraction of streaming multiprocessors doing useful work, and DCGM_FI_DEV_FB_USED, the VRAM held. They diverge constantly. A model server holding a 70B checkpoint while it waits on a slow upstream call shows near-full memory and near-zero SM activity, while a retrieval workload shows the reverse. Sustained throughput needs both high.

Why nvidia-smi can show high utilization on a mostly idle GPU

The headline GPU-Util column reports the share of sample intervals, configurable and often one second, in which at least one kernel ran on the device. A kernel that spins on a spinlock for microseconds and then leaves the device idle for the rest of the window still marks that interval as active. Teams that trust that number size nodes generously and call the fleet healthy.

How to measure GPU utilization in Kubernetes

You need per-pod attribution, not a device-level rollup. The node tells you a GPU is idle. It does not tell you which team reserved sixty percent of that node and left it there.

The DCGM metrics that matter

DCGM Exporter runs as a DaemonSet on every GPU node, scrapes into Prometheus, and graphs in Grafana. Three metrics cover most diagnosis:

• DCGM_FI_PROF_SM_ACTIVE: SM cycles with at least one warp resident. The closest thing to “is the GPU doing real compute.” DCGM_FI_DEV_GPU_UTIL is the coarse figure from nvidia-smi, so track it as a trend line and never as a target.
• DCGM_FI_DEV_MEM_COPY_UTIL: activity of the memory-copy engines. High marks on an inference pod usually mean the model spends its time on host-to-device and device-to-host transfers.
• DCGM_FI_DEV_FB_USED: framebuffer bytes in use. Divide by device memory to see unused VRAM.

Label these with pod and namespace through the exporter’s pod mapping, then group by workload. A cluster-wide p95 hides the namespaces that matter.

Set a baseline before you optimize

Measure each workload’s p95 real usage over a representative window, typically two weeks for inference and a full training run for batch. A cluster average cannot tell a badly sized job from one that bursts on a schedule. Use it to check whether a fix worked. After right-sizing, the same VRAM and SM_ACTIVE shape should hold on a smaller node rather than fall toward zero. After enabling sharing, one device should carry several workloads whose draw adds up instead of one workload using a sliver of it. If neither moves, request size was not your cause and the next fix on the list is.

Why GPU utilization is low: 5 causes

Most underuse lands in one of five buckets, each with a matching fix below. The 2026 report finds the same patterns across customers: capacity is “sized for peak demand rather than typical load,” usage swings materially within a day and between weekdays and weekends, and GPU sharing is “nearly absent in practice.”

1. Oversized requests. A pod asks for nvidia.com/gpu: 1 because that is the smallest schedulable unit. It then uses a fraction of the device, and the scheduler refuses to place anything else on it.
2. One pod per GPU. Kubernetes treats whole GPUs as count-based resources by default. Every replica claims a device even when the model fits in a slice.
3. Poor bin-packing. Workloads land across nodes in ways that leave fragmented headroom. Node A runs one GPU pod, node B runs another, and neither node can be consolidated.
4. Static node pools. GPU nodes stay provisioned through the night and through weekends because the pool does not scale down. Off-peak demand is the largest idle window most teams have.
5. No batch queue. Training and batch jobs run ad hoc. When a batch job finishes early, the node sits idle until the next checkpoint or the next human command.

How to increase GPU utilization: 5 fixes

1. Right-size GPU requests and limits

Right-sizing means setting requests from measured p95 consumption, not from the smallest schedulable unit. Derive it with a Prometheus recording rule taking the 95th percentile of DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_DEV_FB_USED, grouped by pod, over the window. GPU counts only drop when the workload fits a share, but the surrounding CPU, memory, and ephemeral storage requests come down as soon as the data supports it, which lets the pod land on a smaller node.

# convention (padded)
requests:
  cpu: "16"
  memory: 96Gi
  nvidia.com/gpu: 1
# measured p95
requests:
  cpu: "5"
  memory: 32Gi
  nvidia.com/gpu: 1

Start with workloads whose p95 framebuffer usage sits well below device memory; those are the sharing candidates below. For everyone else, the win is freeing CPU and memory so the pod packs into a smaller node. In Cast AI’s production customers, automated rightsizing cuts the provisioned CPU footprint by roughly 50% on average.

Reliability usually improves alongside the footprint. One customer cluster averaged 40 to 50 OOM kills per interval before rightsizing, with spikes above 80. Within days of activation it dropped to near zero, because requests that match real usage also stop the node from evicting pods under memory pressure.

2. Share GPUs: MIG, MPS, and time-slicing

For inference this is the biggest single lever. Compare GPU sharing in Kubernetes options by their trade-offs between isolation and density.

MIG partitions a GPU into hardware-isolated instances, each with its own memory and compute slice, so one tenant cannot stall another. Host-level bottlenecks shared outside the slice, PCIe and copy engines among them, still apply. The cost is fixed geometry: slice sizes are set at device profile config time and resizing means draining pods. The seven-instance maximum holds on A100 40GB, A100 80GB, and H100 class parts, and smaller accelerators cap lower. A practical mix is usually 2 to 4 slices, because larger models need more memory per slice.

MPS runs concurrent CUDA contexts through a shared daemon, with soft isolation only: contexts share memory bandwidth, so one kernel that saturates the bus degrades the others. Density runs higher than MIG and partition sizes stay flexible. MIG and MPS are separate operator modes, so switching between them means a drain and a plugin restart.

Time-slicing multiplexes whole-GPU contexts across time slots with no memory or compute isolation, so it suits dev clusters and bursty batch jobs that tolerate contention. Anything with a latency SLO should use MIG or a whole device.

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4

Apply the ConfigMap, reference it from the ClusterPolicy, and pods requesting nvidia.com/gpu: 1 share the device. MIG takes a different path: a default-mig-parted-config ConfigMap sets profiles such as all-1g.10gb or a mix of two 1g.10gb plus one 3g.40gb, and changing it requires draining the node. We covered the operator mechanics and scheduling implications in our guide to GPU optimization in Kubernetes.

3. Bin-pack with GPU-aware scheduling

Default scheduling treats GPUs as opaque counts: two free GPUs look the same whether they sit on one NUMA domain or across an NVLink island. GPU-aware scheduling adds topology. Our guide to GPU scheduling and bin-packing in Kubernetes explains the placement strategies behind this approach.

Dynamic Resource Allocation (DRA) is the Kubernetes-native direction for this, generally available in Kubernetes 1.34 and not yet a production default across vendor stacks. Each workload declares a ResourceClaim with device constraints, and the scheduler picks a node that satisfies them. Vendor support, driver maturity, and scheduler integration still vary, and the structured-parameters path only works on a node whose NVIDIA driver supports DRA claims, so check the driver version before you plan the upgrade. Consolidating autoscaling sits on top, so a half-empty GPU node merges into another rather than idling through the day.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-claim
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com

One pod holding a GPU on each of two nodes strands both machines. At 20 nodes you tolerate it. Past 500, stranded GPUs compound and the bill shows it.

4. Autoscale GPU nodes to match demand

Autoscaling removes the off-peak floor, and it only reclaims capacity that bin-packing has already consolidated. Two half-empty GPU nodes sitting apart leave scale-down nothing to remove. Karpenter and the cluster autoscaler scale pools down as demand drops and up as pods go pending. The Horizontal Pod Autoscaler handles replicas and reaches GPU metrics only through an external-metrics adapter, such as the Prometheus adapter reading DCGM series.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 15m

Spot is the obvious way to cut the price of the nodes you do need, and almost nobody uses it. Fewer than 2% of the GPUs in the report’s sample run on Spot capacity. Three things decide whether Spot works for yours. Scale-up adds minutes per burst, because a cold pool pays for instance launch, driver initialization, and CUDA warmup. Spot availability varies by accelerator family and region, so pair the on-demand buffer for latency-critical inference with a PodDisruptionBudget, and put instance-family constraints in the NodePool’s spec.template.spec.requirements rather than the Pod spec. Consolidation only fires when requests are accurate, so validate them before switching from WhenEmpty to the underutilized policy.

Our guide to Kubernetes GPU autoscaling covers node scaling, workload scaling, and scale-to-zero for inference. Accurate requests remain essential: an autoscaler bins at request levels, so oversized requests defeat it however fast it reacts.

5. Queue batch and training jobs

Autoscaling matches capacity to demand; queueing matches jobs to free slots. Kueue sits in front of GPU pools as a Kubernetes-native job queue, holding workloads until a quota slot frees up, then admitting them. A ClusterQueue holds shared GPU quota and each namespace gets a LocalQueue pointing at it. How much density this buys depends on the scheduler stack you already run: until Kueue lines up with a gang-scheduling plugin, the gain is smaller.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-cq
spec:
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: default
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: 8

A LocalQueue in each namespace names gpu-cq in spec.clusterQueue; that reference is the only link between a namespace and the quota.

A multi-node training job needs every worker scheduled before any starts, or launched workers hold GPUs and wait. Kueue plus gang scheduling keeps the pool dense and stops partial allocations fragmenting the fleet. See our GPU job queueing with Kueue and DRA for the cost math.

Which fix fits your workload

Automating GPU utilization at scale

Every fix above decays without a loop that re-applies it. Requests drift upward, new workloads arrive with defaults, and the p95 that justified a share of a device shifts within a quarter. The loop is four steps: measure per workload, right-size from measured p95, re-pack onto the smallest node set that holds the load, re-scale the pools to what remains. Run them by hand and each step goes stale at its own rate, which is why manual upkeep fails at the handoff points, where one team quotes last quarter’s p95 and another schedules without knowing the topology. It has to run continuously and per workload, because a handful of namespaces dominate most clusters and a fleet average hides them. That is what Cast AI automates: it reads live utilization and workload requests, tightens GPU, CPU, and memory requests to measured p95, and packs workloads onto the smallest node set that holds them, continuously, instead of once a quarter. Our guide to GPU cost monitoring and attribution explains how to connect utilization with spend per workload and team, and if you’re comparing tooling, the Kubernetes cost optimization tools comparison covers the categories.

See how Cast AI automates GPU optimization.

Frequently asked questions