GPU Scheduling and Bin-Packing in Kubernetes: Pack More AI onto Every GPU

GPU scheduling decides which GPU a pod lands on. Bin-packing decides how densely GPUs fill up before new ones are added. Default Kubernetes handles neither well, which is a primary reason GPU utilization averages just 5% across production clusters. GPU-aware bin-packing, the NVIDIA device plugin, and Dynamic Resource Allocation place more AI work on fewer GPUs. This guide explains how each layer works and how to apply them.

Key Takeaways

• According to the Cast AI 2026 State of Kubernetes Optimization Report, average GPU utilization across tens of thousands of production clusters is 5%. CPU averages 8%, memory 20%.
• The default scheduler uses LeastAllocated scoring, which spreads pods across many nodes instead of filling existing ones first.
• The NVIDIA device plugin exposes GPUs as integer resources (nvidia.com/gpu). It has no attribute awareness and no fractional allocation support.
• MIG partitions one H100 into up to 7 hardware-isolated instances. Time-slicing creates up to 48 virtual replicas through CUDA context switching.
• GPU-aware bin-packing requires three technical layers: right-sized requests, MostAllocated scheduler scoring, and ongoing consolidation.
• Dynamic Resource Allocation (DRA), stable in Kubernetes 1.34, replaces integer counting with attribute-based CEL filtering and supports device sharing across pods.

How Kubernetes Schedules GPUs

The Device Plugin Framework

Kubernetes does not manage GPUs natively. Instead, it relies on the device plugin framework, which reached stable status in Kubernetes 1.26. Hardware vendors, including NVIDIA, AMD, and Intel, deploy DaemonSet-based plugins that register with the kubelet on each node. Each plugin advertises its devices as extended resources. For NVIDIA GPUs, that resource is nvidia.com/gpu.

A pod requests a GPU with a single field in its spec:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-job
spec:
  containers:
  - name: trainer
    image: nvcr.io/nvidia/pytorch:24.01-py3
    resources:
      limits:
        nvidia.com/gpu: "1"

The scheduler sees an integer counter. It finds a node with at least one available nvidia.com/gpu unit and places the pod there. That is the complete default GPU scheduling logic.

The NVIDIA GPU Operator

The NVIDIA GPU Operator automates the full driver and plugin stack. It installs GPU drivers, the device plugin, the DCGM Exporter for metrics, the MIG Manager DaemonSet for partition management, and GPU Feature Discovery for node labeling. You deploy it via a single Helm chart.

Four DCGM metrics matter most for utilization monitoring:

• DCGM_FI_DEV_GPU_UTIL: compute utilization percentage
• DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE: framebuffer memory usage and headroom
• DCGM_FI_DEV_SM_CLOCK: streaming multiprocessor clock, which drops under thermal throttling
• DCGM_FI_DEV_POWER_USAGE: power draw

These metrics show what the GPU actually does, not just what Kubernetes believes it is doing. That distinction matters for consolidation decisions.

Why Default Scheduling Wastes GPUs

LeastAllocated: The Spreading Problem

The default scheduler scores nodes using LeastAllocated. It assigns higher scores to nodes with more free capacity. As a result, new pods consistently land on the emptiest available node. The effect is predictable: workloads spread thin across many nodes, and no single node fills up.

For CPU workloads, this is a reasonable trade-off. For GPUs, it is expensive. An H100 on AWS costs roughly $6.50 to $7.00 per GPU per hour on demand. GCP pricing runs $9.00 to $11.50. Azure runs $11.00 to $13.00. Every half-empty node burns that rate with no return. An idle H100 costs approximately $4,954 per month.

The Binary Allocation Problem

The scheduler allocates based on requests, not actual usage. A pod requesting one GPU gets exclusive ownership of the entire physical device, regardless of whether it uses 10% or 90% of the hardware. There is no native fractional allocation. There is no attribute awareness. The scheduler cannot distinguish an A100 from an H100. It only counts.

The Cast AI 2026 State of Kubernetes Optimization Report measured this problem across tens of thousands of production clusters: average GPU utilization is 5%. Laurent Gil, co-founder and president of Cast AI, summarizes it directly: “A GPU sitting idle costs dollars per hour. A CPU sitting idle costs cents. And 95% of GPU capacity is doing nothing.”

The best-measured cluster in the report, a 136-node H200 LLM inference fleet, reached 49% average utilization. The fleet-wide average is 5%. That 10x gap comes almost entirely from technique, not from hardware differences.

GPU-Aware Bin-Packing

Prerequisite: Right-Size GPU Requests

Right-sizing resource requests is a prerequisite for every layer that follows. If requests are too large, nodes appear full while their GPUs sit idle. The scheduler allocates based on what pods declare, not what they consume.

Use kubectl top pods --containers to identify pods consuming far less than they declare. For VRAM specifically, query DCGM_FI_DEV_FB_USED to see actual framebuffer consumption. Pods that declare a full GPU but consume 8 GB of an 80 GB H100 are prime candidates for MIG partitioning or time-slicing.

Layer 1: Scheduler Scoring

Switch the scheduler scoring plugin from LeastAllocated to MostAllocated. This tells the scheduler to prefer the most utilized node available, so new pods fill existing nodes before the cluster expands. The resources: field is required. Without it, the scheduler weighs CPU and memory equally with GPUs and GPU density does not drive placement.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default-scheduler
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated
        resources:
        - name: cpu
          weight: 1
        - name: memory
          weight: 1
        - name: nvidia.com/gpu
          weight: 3

The higher weight for nvidia.com/gpu ensures GPU bin-packing drives scheduling decisions. A node with a busy GPU scores higher than a node with busy CPU, which matches the economics of GPU infrastructure.

Layer 2: Ongoing Consolidation

Changing the scheduler fills new nodes more densely. It does not reclaim capacity from nodes already sitting underutilized. A consolidation controller handles that: it identifies underutilized nodes, migrates their pods, and removes empty nodes.

Before enabling consolidation, ensure all production workloads have PodDisruptionBudgets configured. Cast AI Evictor and Karpenter both respect PDBs. A pod with no PDB can be evicted immediately during consolidation, terminating an in-progress training run without warning. A minimal PDB for an inference deployment:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: inference-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: inference

GPU node cold starts take 3 to 8 minutes on instances like AWS p4d and p5, including driver initialization and NVLink fabric bring-up. Aggressive consolidation that frequently terminates GPU nodes creates scheduling delays of that duration. Set consolidation thresholds to balance empty-node savings against cold-start cost. Cast AI Evictor pre-provisions replacement capacity before draining, reducing cold-start exposure.

For Karpenter on EKS, the WhenUnderutilized policy works without a consolidateAfter field:

disruption:
  consolidationPolicy: WhenUnderutilized
  budgets:
  - nodes: "10%"

Note that consolidateAfter is only valid with the WhenEmpty policy. With WhenEmpty, Karpenter waits the specified duration after a node becomes fully empty before terminating it:

disruption:
  consolidationPolicy: WhenEmpty
  consolidateAfter: 30m

Using consolidateAfter with WhenUnderutilized is invalid and will cause the NodePool to be rejected at admission. For GPU fleets, WhenUnderutilized with a 10% node budget is the recommended starting point.

For the complete consolidation approach, see the Kubernetes bin-packing guide.

MIG: Hardware-Level Partitioning

MIG (Multi-Instance GPU) partitions a single physical GPU into up to 7 isolated instances at the hardware level. Each MIG instance gets dedicated compute cores, dedicated memory bandwidth, and a dedicated L2 cache. Workloads in separate instances cannot interfere with each other. MIG is supported on A100, A30, H100, and H200.

H100 MIG profiles, from smallest to largest:

• 1g.10gb: 7 instances, 10 GB each
• 2g.20gb: 3 instances, 20 GB each
• 3g.40gb: 2 instances, 40 GB each
• 7g.80gb: 1 instance, full GPU

Time-Slicing: Software-Level Sharing

Time-slicing creates virtual GPU replicas through CUDA context switching. Up to 48 virtual replicas per physical GPU are supported. It works on any CUDA-capable GPU, making it useful for older hardware that does not support MIG.

For vLLM deployments, set --gpu-memory-utilization per replica. With 4 replicas sharing one GPU, use 0.22 rather than the arithmetic result of 0.25. The lower value reserves headroom for CUDA context overhead, which runs approximately 1 to 2 GB per context. Adjust the value based on your model’s actual VRAM footprint.

Time-slicing has no memory enforcement boundary. Two pods each configured for 0.22 can simultaneously allocate VRAM, causing OOM failures that kill all co-tenants on the GPU. The --gpu-memory-utilization flag is an application-level soft limit only. It does not prevent another replica from allocating memory at the same time. Use MIG when memory isolation is a production requirement.

MIG vs. Time-Slicing: When to Use Which

For detailed guidance on combining both approaches for maximum density, see the GPU sharing for Kubernetes cost optimization guide.

The Economics

Four developers sharing one H100 at $6.88 per hour costs $1.72 per developer per hour. Add Spot pricing at a 60% discount and the cost drops to roughly $0.69 per developer per hour. That is approximately a 90% reduction compared to one dedicated H100 per developer. Despite this, fewer than 2% of enterprise Kubernetes GPU workloads currently run on Spot, even though AWS Spot saves 60 to 91%, GCP 60 to 80%, and Azure 60 to 90% compared to on-demand rates.

Dynamic Resource Allocation

What DRA Solves

Dynamic Resource Allocation (DRA) is the Kubernetes API designed to replace the device plugin’s opaque integer model. DRA reached GA (stable) in Kubernetes 1.34 and is enabled by default in Kubernetes 1.35. AWS EKS supports DRA from Kubernetes 1.33. The API group is resource.k8s.io/v1.

DRA introduces four key objects:

• ResourceSlice: the driver publishes available device inventory to the API server
• DeviceClass: a cluster-wide policy defining a class of devices and their default configuration
• ResourceClaimTemplate: a per-namespace template for generating ResourceClaims
• ResourceClaim: the actual per-pod binding to a specific device

Attribute-Based Scheduling with CEL

With DRA, pods select devices using CEL (Common Expression Language) expressions rather than integer counts. CEL is a lightweight expression language embedded in Kubernetes that evaluates device attributes at scheduling time. Instead of asking for “one GPU,” a pod asks for a GPU with specific architecture and minimum memory.

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: hopper-gpu-claim
spec:
  devices:
    requests:
    - name: gpu
      deviceClassName: gpu.nvidia.com
      selectors:
      - cel:
          expression: >
            device.attributes['gpu.nvidia.com'].architecture == 'hopper' &&
            device.attributes['gpu.nvidia.com'].memory.isGreaterThan(quantity('40Gi'))

This ResourceClaim selects only Hopper-architecture GPUs with more than 40 GB of memory. The scheduler evaluates the CEL expression against the ResourceSlice inventory and places the pod on a matching device. No more opaque counters. No more platform-blind scheduling.

Default vs GPU-Aware vs DRA: Side-by-Side

DRA is not a drop-in replacement for all workloads today. Device plugins remain stable and widely deployed. For new clusters on Kubernetes 1.34 or later, however, DRA is the forward path.

How Cast AI Automates GPU Scheduling and Bin-Packing

Three Layers Plus One

Manual implementation of the three technical layers requires configuration, ongoing maintenance, and cross-cluster coordination. MIG profiles change as workloads change. Scheduler scoring plugins need deployment and monitoring. Consolidation needs active eviction logic with cold-start awareness built in. At 50 nodes, this is manageable. At 500 nodes, it is a full-time job.

Cast AI automates all three layers and adds a fourth. The three technical layers are: MostAllocated scheduler scoring with GPU-weighted resources deployed cluster-wide; GPU sharing via MIG partition management and time-slicing replica configuration; and continuous consolidation through the Cast AI Evictor, which verifies spare capacity using actual DCGM metrics rather than request declarations, migrates pods while respecting PodDisruptionBudgets, and removes empty nodes.

The fourth layer is automated GPU node lifecycle: scale-to-zero for idle GPU node pools, pre-provisioning replacement capacity before draining nodes to avoid cold-start delays, and predictive scaling based on workload patterns. This runs across AWS, GCP, and Azure without separate per-cloud configuration.

ALLEN Digital combined time-slicing, Spot instances, and Cast AI bin-packing consolidation to achieve over 70% total savings compared to their prior SageMaker setup. That result comes from stacking all four layers, not from applying any single one in isolation.

An idle H100 costs roughly $4,954 per month. Recovering even one idle GPU per week across a mid-size fleet adds up fast. For the complete GPU optimization framework, including DCGM alerting thresholds and MIG configuration templates, see the Kubernetes GPU optimization guide.

Conclusion

GPU scheduling and bin-packing are not configuration details. They are the operational difference between 5% utilization and the 49% achievable with the right techniques. The default scheduler spreads pods thin. The device plugin cannot see GPU attributes. Neither problem requires new hardware to solve.

Fixing it requires three technical layers. First, switch scheduler scoring to MostAllocated with GPU-weighted resources. Second, add GPU sharing via MIG or time-slicing, choosing based on whether memory isolation is required. Third, run continuous consolidation with PDB protection and cold-start awareness in place.

Cast AI adds the fourth layer: automated GPU node lifecycle that removes the operational burden of managing cold-start windows, pre-provisioning, and scale-to-zero across multiple clouds. Each step recovers real dollars at real GPU prices.

For the full Kubernetes GPU optimization framework, start with the Kubernetes GPU optimization guide.

Frequently Asked Questions