GPU sharing and GPU Rightsizing: Stop Wasting GPU

Key takeaways

• Average GPU utilization in production Kubernetes clusters is just 5%, according to the Cast AI 2026 State of Kubernetes Optimization Report.
• GPU sharing, via time-slicing, MIG or MPS, let multiple pods share one physical GPU.
• Time-slicing is software-based sharing through context switching and works on any NVIDIA GPU. MIG provides hardware-level memory and fault isolation. MPS enables truly concurrent kernel execution with isolated address spaces but limited fault protection, best for trusted same-tenant workloads.
• GPU rightsizing matches resource requests to actual utilization, eliminating waste on capacity workloads never consume.
• Cast AI automates the full rightsizing loop: measure utilization, generate recommendations, and apply corrected requests automatically.

Why whole-GPU allocation wastes money

Kubernetes GPU scheduling is all-or-nothing by default. A pod requests nvidia.com/gpu: 1 and the scheduler pins the entire physical card to that pod. No other workload can touch it, even when the pod is idle.

In production, most inference pods consume a tiny fraction of the GPU they hold. Average utilization across production clusters is just 5%, based on data from tens of thousands of clusters on AWS, GCP, and Azure. That figure comes directly from the Cast AI 2026 State of Kubernetes Optimization Report. In other words, 95 cents of every GPU dollar goes to idle silicon. At AWS Capacity Block pricing for H100 instances (~$6.88 per GPU-hour), a 20-GPU cluster running 720 hours per month at 5% utilization wastes roughly $94,000 per month in idle GPU capacity.

The economics are getting harder, not easier. H200 AWS Capacity Block pricing increased 15% in January 2026, the first GPU price increase in roughly two decades, according to the Cast AI 2026 State of Kubernetes Optimization Report. Teams scaling AI workloads pay more per GPU while using less of each GPU. That is a compounding problem with no self-correcting mechanism.

The gap is technique, not hardware. The best-performing cluster in the Cast AI dataset is a 136-node H200 fleet running at 49% utilization, roughly 10x the average. Those teams apply GPU sharing and rightsizing. The hardware is the same; the configuration is different.

GPU sharing: time-slicing, MIG, and MPS

For a broader look at GPU sharing strategies, see the Cast AI guide on GPU sharing in Kubernetes for cost optimization.

Three GPU sharing techniques: time-slicing, Multi-Instance GPU (MIG), and Multi-Process Service (MPS) – Cast AI supports all three through node templates and automated node provisioning — no manual GPU Operator ConfigMaps or node labeling required. For a complete overview of supported sharing methods, provider compatibility, and a comparison table, see the Cast AI GPU sharing documentation. Cast AI also supports GPU sharing through Dynamic Resource Allocation (DRA) on Kubernetes 1.34+.

Time-slicing: sharing through context switching

Time-slicing allows multiple workloads to share a single physical GPU by rapidly switching between processes. GPU time-slicing is achieved through rapid context switching, where each process gets an equal share of GPU time, compute resources are assigned to one process at a time, and GPU memory is shared between all processes. Time-slicing works on any NVIDIA GPU.

Configure time-slicing in a Cast AI node template. In the Cast AI console, create or edit a node template, enable GPU time-slicing, and configure sharing parameters:

• Default shared clients per GPU: the default number of workloads that can share each GPU (1–48)
• Sharing configuration per GPU type: override defaults for specific GPU models

Or with Terraform:

resource "castai_node_template" "gpu-timeslicing" {
  # ... other configuration

  gpu {
    enable_time_sharing             = true
    default_shared_clients_per_gpu  = 4

    sharing_configuration = {
      "nvidia-tesla-t4" = {
        shared_clients_per_gpu = 4
      }
      "nvidia-tesla-a100" = {
        shared_clients_per_gpu = 8
      }
    }
  }
}

When using GPU time-slicing, pods continue to request GPUs using the standard nvidia.com/gpu resource — no changes to pod specifications are required.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-timeslicing-workload
spec:
  replicas: 10  # Can schedule 10 pods on a single GPU with 10x sharing
  selector:
    matchLabels:
      app: gpu-timeslicing-workload
  template:
    metadata:
      labels:
        app: gpu-timeslicing-workload
    spec:
      containers:
      - name: gpu-workload
        image: your-gpu-image
        resources:
          limits:
            nvidia.com/gpu: 1

Cast AI calculates extended GPU capacity as GPU_COUNT * SHARED_CLIENTS_PER_GPU. At 4 shared clients, each workload gets an equal 25% share of GPU time. If actual compute demand is spread across all 4 pods simultaneously, effective per-workload GPU cost drops by roughly 75%. If all pods run at peak simultaneously, they compete for the same time slots, but idle inference pods still benefit fully from the cost sharing.

With Dynamic Resource Allocation (DRA) on Kubernetes 1.34+, time-slicing is configured in a shared ResourceClaim instead of a node template:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: gpu-timeslicing-shared
  namespace: gpu-dra-example
spec:
  devices:
    requests:
    - name: ts-gpu
      exactly:
        deviceClassName: gpu.nvidia.com
    config:
    - requests: ["ts-gpu"]
      opaque:
        driver: gpu.nvidia.com
        parameters:
          apiVersion: resource.nvidia.com/v1beta1
          kind: GpuConfig
          sharing:
            strategy: TimeSlicing
            timeSlicingConfig:
              interval: Long          # Default | Short | Medium | Long

For full Deployment examples, see the Cast AI DRA documentation and the Deploying GPU workloads with DRA blog post.

MIG: hardware-level partitions

MIG is a hardware feature available on NVIDIA GPUs starting with the Ampere architecture (compute capability >= 8.0), including A100, A30, H100, H200, B200, and GB200. It partitions the physical GPU into isolated slices at the silicon level. Each slice gets dedicated memory, cache, and Streaming Multiprocessors (SM).

A single H100 80GB supports up to 7 MIG instances using the 1g.10gb profile, giving each slice 10 GB of memory. The H200 141GB supports 7 instances at 1g.18gb, providing 18 GB per slice. The table below covers the smallest available profile and maximum instance count for each MIG-capable GPU:

MIG is also supported on the H20 (96 GB, 7 instances) and GB200 (186 GB, 7 instances). See the NVIDIA MIG User Guide for the full list, including workstation-class RTX PRO GPUs.

Cast AI automates MIG provisioning. No ConfigMaps or node labeling are required — the autoscaler detects the partition size from the workload’s node selector, provisions a MIG-capable node, and configures the GPU partitions automatically.

On GKE, the partition size goes in the node selector value:

nodeSelector:
  cloud.google.com/gke-gpu-partition-size: 1g.5gb

On EKS, the partition size is embedded in the node selector key:

nodeSelector:
  nvidia.com/gpu.mig-partition-1g.5gb: "true"

A complete MIG workload deployment targeting 7 1g.5gb partitions on GKE:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mig-workload
spec:
  replicas: 7  # One pod per MIG partition
  selector:
    matchLabels:
      app: mig-workload
  template:
    metadata:
      labels:
        app: mig-workload
    spec:
      nodeSelector:
        cloud.google.com/gke-gpu-partition-size: 1g.5gb
      tolerations:
        - key: "nvidia.com/gpu.mig"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
      - name: gpu-workload
        image: your-gpu-image
        resources:
          limits:
            nvidia.com/gpu: 1

On EKS, swap the node selector to nvidia.com/gpu.mig-partition-1g.5gb: "true". Pods request nvidia.com/gpu: 1 — Cast AI treats each MIG partition as a separate schedulable GPU.

You can also target specific GPU models:

nodeSelector:
  cloud.google.com/gke-gpu-partition-size: 1g.5gb
  cloud.google.com/gke-accelerator: "nvidia-tesla-a100"

Verify MIG-enabled nodes and available partitions:

# List MIG-enabled nodes
kubectl get nodes -l nvidia.com/gpu.mig=true

# Check available partitions
kubectl describe node <node-name> | grep nvidia.com/gpu

MIG can also be combined with time-slicing for maximum density. A single A100 with 7 1g.5gb partitions and 4x time-slicing supports 28 concurrent workloads (7 × 4 = 28). See the Cast AI MIG documentation for details.

With DRA on Kubernetes 1.34+, each pod gets a dedicated MIG partition via a ResourceClaimTemplate:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-mig-template
  namespace: gpu-dra-example
spec:
  spec:
    devices:
      requests:
      - name: mig-1g-5gb
        exactly:
          deviceClassName: mig.nvidia.com
          selectors:
          - cel:
              expression: |
                device.attributes["gpu.nvidia.com"].profile == "1g.5gb"

MPS: concurrent kernel execution

NVIDIA Multi-Process Service (MPS) is a lightweight runtime service that enables multiple CUDA applications to share a single GPU simultaneously. Unlike time-slicing, which rapidly switches execution between processes, MPS allows truly concurrent execution of GPU kernels from different processes through a client-server architecture, where the MPS server shares a single set of GPU scheduling resources across all clients. The MPS client runtime is built into the standard CUDA driver, so MPS is transparent to CUDA applications — no code changes or recompilation are required.

MPS clients have fully isolated GPU address spaces — each client owns its own address space, independent of other clients. However, MPS provides only limited fault containment: a fatal GPU fault in one client can affect other clients sharing the same GPU. On Volta and newer GPUs, the MPS server can recover after faulting clients disconnect; on pre-Volta GPUs, a fatal fault shuts down the entire MPS server. This limited fault isolation makes MPS suitable for trusted, same-tenant workloads where strict MIG isolation is not required but concurrent execution improves throughput for compute-bound jobs. On Volta and newer GPUs (compute capability ≥ 7.0), no changes to pod specifications are required beyond standard GPU resource requests. On pre-Volta GPUs, pods must set hostIPC: true to communicate with the MPS control daemon via UNIX domain sockets.

Pods continue to request nvidia.com/gpu: 1 — no special resource name is needed.

Configure MPS with Terraform — set the GPU sharing strategy to mps, configure the default shared clients per GPU, and optionally override the count per GPU type:

resource "castai_node_template" "gpu-mps" {
  # ... other configuration

  gpu {
    sharing_strategy               = "mps"
    default_shared_clients_per_gpu = 4

    sharing_configuration = {
      "nvidia-tesla-t4" = {
        shared_clients_per_gpu = 4
      }
      "nvidia-tesla-a100" = {
        shared_clients_per_gpu = 8
      }
    }
  }
}

A complete MPS workload deployment targeting 4 pods on a single GPU with default_shared_clients_per_gpu = 4:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-mps-workload
spec:
  replicas: 4  # 4 pods share one GPU via MPS
  selector:
    matchLabels:
      app: gpu-mps-workload
  template:
    metadata:
      labels:
        app: gpu-mps-workload
    spec:
      containers:
      - name: gpu-workload
        image: your-gpu-image
        resources:
          limits:
            nvidia.com/gpu: 1

With DRA on Kubernetes 1.34+, MPS is configured in a shared ResourceClaim:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: gpu-mps-shared
  namespace: gpu-dra-example
spec:
  devices:
    requests:
    - name: mps-gpu
      exactly:
        deviceClassName: gpu.nvidia.com
    config:
    - requests: ["mps-gpu"]
      opaque:
        driver: gpu.nvidia.com
        parameters:
          apiVersion: resource.nvidia.com/v1beta1
          kind: GpuConfig
          sharing:
            strategy: MPS
            mpsConfig:
              defaultActiveThreadPercentage: 33
              defaultPinnedDeviceMemoryLimit: 5Gi

For full Deployment examples, see the Cast AI DRA documentation and the Deploying GPU workloads with DRA blog post.

Time-slicing vs. MIG vs. MPS: corrected comparison

GPU rightsizing: match requests to real usage

GPU sharing reduces allocation granularity. Rightsizing ensures your requests actually match workload behavior. Together, they eliminate two distinct kinds of waste: holding a whole card when you need a fraction, and requesting a large fraction when you need a small one.

Measuring with DCGM metrics

The NVIDIA DCGM Exporter exposes the metrics you need. Two are essential for rightsizing decisions:

DCGM_FI_DEV_GPU_UTIL  # GPU compute utilization (%)
DCGM_FI_DEV_FB_USED   # Framebuffer memory used (MiB)

Use these PromQL queries to establish the utilization baselines you need for rightsizing decisions:

# 7-day average GPU compute utilization (5-minute resolution steps)
avg_over_time(DCGM_FI_DEV_GPU_UTIL[7d:5m])

# Peak GPU memory used over the observation window
max_over_time(DCGM_FI_DEV_FB_USED[7d:5m])

Collect enough data to capture your typical traffic peaks before drawing rightsizing conclusions. Then compare observed peaks against what each pod’s resource requests specify.

In most production clusters, pods request a full GPU but peak at 5–10% compute utilization and use 20–30% of GPU memory. That discrepancy is the rightsizing opportunity: most pods could run on a single MIG slice and still have headroom.

Setting corrected resource requests

After measuring, set GPU requests to match observed peak usage with a reasonable buffer. For memory, base your request on the peak (P100) of DCGM_FI_DEV_FB_USED plus 20% headroom. For compute, GPU requests in Kubernetes are integers, so rightsizing often means switching to a smaller MIG profile rather than tuning a fractional value.

One important caveat: GPU resource changes require a pod restart. Plan rightsizing during deployment windows or maintenance cycles.

How Cast AI automates GPU rightsizing

Manual rightsizing across hundreds of workloads is operationally expensive. The Cast AI Workload Autoscaler integrates DCGM Exporter metrics via the Kvisor agent. It continuously measures real GPU and memory utilization, generates rightsizing recommendations, and applies them automatically without manual intervention.

The Workload Autoscaler also bins workloads tightly onto fewer nodes, compounding the efficiency gains. Teams using Cast AI for automated workload rightsizing have moved GPU utilization from the 5% cluster average into the 40–60% range across production clusters, based on Cast AI telemetry from its 2026 State of Kubernetes Optimization dataset. For the full picture on Kubernetes GPU optimization, the Cast AI blog covers the complete stack from scheduling to bin-packing.

When to use which: inference vs. training

Choosing between time-slicing, MIG, MPS, and GPU rightsizing depends on workload type and isolation requirements. The following table provides direct guidance:

Conclusion

If your hardware is older, time-slicing gets you GPU sharing immediately, without waiting for your next hardware refresh. If your cluster runs A100, H100, or H200 nodes, start with MIG. It gives you hardware-level isolation and makes GPU sharing safe for production multi-tenant inference. For trusted, same-tenant compute-bound workloads on Volta+ GPUs, MPS offers the best throughput.

In all cases, begin with rightsizing. It requires no hardware constraints and delivers savings on day one. Measure DCGM utilization over a sufficient period, set corrected requests, then layer GPU sharing based on workload isolation needs. Tighter requests plus shared physical GPUs compound together: that combination is where the largest efficiency gains appear.

Cast AI handles the rightsizing loop automatically. The Workload Autoscaler monitors GPU utilization via DCGM, recommends corrected requests, and applies them without manual intervention. It also bins the rightsized workloads tightly, multiplying the effect across your fleet. For production benchmark data, start with the Cast AI 2026 State of Kubernetes Optimization Report.

To start: connect your DCGM metrics to Cast AI’s Workload Autoscaler, let it observe utilization data over a sufficient period, and run your first automated rightsizing cycle. Most teams see their first GPU request correction within 48 hours. See how automated GPU optimization can help you turn GPU utilization data into lower infrastructure costs.

Frequently Asked Questions