GPU Cost Monitoring: Track Utilization and Attribute AI Spend

Key takeaways

• Average GPU utilization in Kubernetes sits at just 5%, according to the Cast AI 2026 State of Kubernetes Optimization Report. AKS clusters average 2%, EKS 5%, and GKE 6%.
• An idle H100 GPU on AWS costs approximately $8,850 per GPU per month at p5 on-demand pricing. A team of 20 engineers with idle GPU habits can burn over $177K per month without noticing.
• DCGM (NVIDIA Data Center GPU Manager) is the standard metrics source for GPU utilization in Kubernetes. The dcgm-exporter DaemonSet exposes per-pod, per-namespace Prometheus metrics.
• Two GPU cost attribution models exist: requests-based (charges for what a pod reserves) and usage-based (charges on actual utilization). Each serves a different purpose.
• Idle GPU detection requires combining DCGM_FI_DEV_GPU_UTIL below 20% for 30 minutes with a large DCGM_FI_DEV_FB_FREE reading on an allocated pod.
• Turning visibility into savings requires automating the response: right-sizing requests, scaling down idle nodes, and setting hard budget guardrails per team.

Why GPU visibility comes first

GPU hardware is the most expensive resource in your cluster. It is also the least utilized. CPU averages 8% utilization across Kubernetes workloads. Memory averages 20%. GPU sits at 5%. That gap is not an accident.

Teams provision GPU nodes for peak demand, then forget to scale them down. Training jobs finish but pods remain running. Jupyter notebooks hold GPU allocations for hours with no active kernel. Additionally, nvidia.com/gpu resource requests behave differently from CPU: one pod claims the whole GPU, even if it uses 3% of compute and 10% of VRAM.

Meanwhile, GPU costs keep rising. According to Cast AI’s GPU Price Report, H200 Capacity Block prices on AWS increased in early 2026, the first GPU price increase since EC2 launched. An idle H100 GPU costs approximately $8,850 per GPU per month on AWS p5 on-demand pricing. For a team of 20 with sloppy GPU habits, that number compounds fast, reaching over $177K per month in wasted spend.

However, you cannot address a cost problem you cannot measure. Kubernetes does not expose GPU utilization natively. Standard CPU and memory metrics from the kubelet give you nothing on what the GPU is actually doing. Therefore, GPU cost monitoring requires a dedicated metrics pipeline, starting with DCGM.

For broader context on how GPU optimization fits into overall Kubernetes cost management, see Kubernetes GPU Optimization.

What to track: utilization, memory, idle, DCGM

DCGM stands for NVIDIA Data Center GPU Manager. It is the standard telemetry layer for datacenter GPUs. In Kubernetes, the dcgm-exporter runs as a DaemonSet, typically deployed as part of the NVIDIA GPU Operator. It connects to the kubelet pod-resources socket and appends pod and namespace labels to every metric it exposes. The result is a Prometheus-compatible /metrics endpoint with per-pod GPU telemetry. Retain DCGM metrics for at least 90 days in your time-series database. Shorter retention windows prevent month-over-month trend analysis and make it harder to validate showback reports before implementing chargeback.

The 8 GPU metrics that matter

Not every DCGM metric is equally useful for cost monitoring. The following eight cover compute utilization, memory pressure, power draw, thermal limits, and hardware health.

Throttle temperatures vary slightly by SKU and form factor (SXM5 vs PCIe). Verify against NVIDIA’s official thermal specifications for your specific GPU model.

Detecting idle GPUs with PromQL

Idle GPU detection works by combining a low compute utilization signal with evidence that a pod still holds the allocation. The following PromQL query identifies GPUs running below 20% utilization over a 30-minute window:

# Flag GPUs with under 20% average utilization over the last 30 minutes
avg_over_time(DCGM_FI_DEV_GPU_UTIL[30m]) < 20

This query returns GPU instances by device. However, it does not yet tell you which pod or team owns the idle GPU. For that, you need label-aware queries scoped to the pod and namespace dimensions that dcgm-exporter appends automatically.

Per-pod GPU utilization over the last 30 minutes:

# GPU utilization per pod, averaged over 30 minutes
avg by (pod, namespace, gpu) (
  avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[30m])
)

Per-namespace GPU utilization, useful for team-level dashboards:

# Average GPU utilization by namespace over the last hour
avg by (namespace) (
  avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[1h])
)

To confirm idle state, add the framebuffer check. An allocated pod with low utilization and high free VRAM is definitively idle:

# Idle GPU: low utilization AND large free VRAM on an allocated pod
(avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[30m]) < 20)
and
(DCGM_FI_DEV_FB_FREE{pod!=""} > 40000)

Cost attribution: per workload, team, app

GPU metrics visibility is only half the job. The other half is answering: who owns this cost? Attribution turns raw utilization numbers into accountability.

The attribution hierarchy

Namespace is the first attribution boundary in Kubernetes. Each team or application typically lives in its own namespace, so namespace-level GPU spend gives you a fast first cut. For more detail, labels extend attribution further. Common label keys include team, app, project, and environment.

For a deeper look at how cost allocation works across all resource types in Kubernetes, see Kubernetes Cost Allocation.

Two attribution models: requests-based vs. usage-based

Two chargeback models suit different goals. First, the requests-based model charges each team for every nvidia.com/gpu resource request in their pods, regardless of actual utilization. This model creates strong accountability: teams pay for what they reserve, not what they use. Therefore, it incentivizes right-sizing requests and releasing GPUs when not needed.

Second, the usage-based model charges on actual GPU utilization. This model is better for COGS calculations and for understanding the real cost of an AI feature or model training run. However, it requires more instrumentation and is harder to explain to teams unfamiliar with GPU metrics.

Most engineering organizations start with requests-based attribution for team accountability, then add usage-based tracking for product COGS. Use both together for a complete picture.

Showback before chargeback

Start with showback: report GPU costs back to teams without billing them. Run showback for 60-90 days to validate your attribution model before implementing financial accountability. Teams need time to see the data, fix mislabeled workloads, and clean up orphaned GPU reservations.

Before activating billing accountability, verify that more than 95% of GPU pod hours carry a team label. Run this check:

# Percentage of GPU pod-hours with team label
(
  count(kube_pod_labels{label_team!=""} * on(pod,namespace) group_left() (DCGM_FI_DEV_GPU_UTIL > 0))
  /
  count(DCGM_FI_DEV_GPU_UTIL > 0)
) * 100

If coverage is below 95%, fix the unlabeled pods before charging teams. An unlabeled GPU pod creates attribution debt that compounds over time.

After showback is validated, choose your billing rate model:

• Unblended rates: charge teams the actual hourly rate paid for each GPU. Best for accuracy.
• Amortized rates: spread Reserved Instance or Savings Plan commitments evenly. Best for team budget predictability.

For most teams, amortized rates produce fairer internal billing. Finance owns the decision. Loop them in before going live.

Commitment timing

Optimize first, then commit. If you purchase Reserved Instances or Savings Plans against unoptimized GPU baselines, you will over-commit. Run your idle detection pipeline for at least one full 30-day cycle. Before committing, confirm that wasted GPU hours fall below 15% of total GPU hours in the trailing 30-day window. If idle waste is still above 15%, continue optimizing before purchasing. Committing above that threshold locks in overprovision at reserved pricing.

The chargeback formula

For usage-based GPU chargeback, the formula is straightforward:

team_gpu_cost = (team_gpu_util / total_gpu_util) × gpu_hour_cost

Before running the chargeback formula, verify two things. First, dcgm-exporter must be version 3.x or later, which appends pod and namespace labels to DCGM metrics automatically via the kubelet pod-resources socket. Second, kube-state-metrics must expose your team labels. By default, kube-state-metrics allowlists only a subset of pod labels. Add your label to the kube-state-metrics --metric-labels-allowlist flag:

--metric-labels-allowlist=pods=[team,app,project,environment]

Without this, kube_pod_labels will not include your team label and the chargeback query will silently return no results.

In PromQL, a per-team chargeback rate looks like this:

# Proportional GPU cost share by team label
sum by (team) (avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[1h]) * on(pod) group_left(team) kube_pod_labels)
/
sum(avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[1h]))
* gpu_hour_cost

Replace gpu_hour_cost with the actual on-demand or reserved instance price for your GPU node type. For an AWS p5.48xlarge (8x H100), divide the per-node hourly rate by 8 to get the per-GPU cost.

Turning visibility into savings

Dashboards and alerts are inputs, not outcomes. Visibility only produces savings when it triggers action. The following steps move you from monitoring to reclaimed spend.

Step 1: Set idle GPU alerts

Configure Prometheus alerting rules to fire when the idle GPU PromQL condition holds for 30 minutes. Route alerts to the owning team’s Slack channel using the namespace or team label. This creates direct feedback: the team that owns the idle pod receives the alert, not a central platform team.

Step 2: Right-size GPU requests

Most GPU pods request whole GPU units when fractional allocation would suffice for inference workloads. Review DCGM_FI_DEV_FB_USED alongside DCGM_FI_DEV_GPU_UTIL for each workload. For example, an inference pod using 8 GB of a 80 GB VRAM GPU and 12% compute is a candidate for multi-instance GPU (MIG) or time-slicing. Additionally, setting proper resource limits prevents pods from holding GPU allocations after their primary task completes.

Step 3: Automate node scale-down

Node-level idle detection closes the loop. When all pods on a GPU node show low utilization and the node has been in that state for a defined window, automated scale-down removes the node and stops billing. However, this requires the cluster autoscaler or an intelligent autoscaling layer to understand GPU node costs and act on utilization signals, not just pod count.

Cast AI connects GPU utilization data directly to autonomous scaling decisions. It reads DCGM metrics, identifies idle GPU nodes, and removes them without manual intervention. For clusters where GPU spend is the dominant cost driver, this closes the gap between what you monitor and what you actually save. See Cast AI GPU Optimization to understand how automated GPU rightsizing works in practice.

Step 4: Set per-team GPU budgets

Attribution without limits creates reporting without accountability. Set namespace-level GPU resource quotas to cap the maximum number of GPUs a team can hold simultaneously. Then, pair quotas with showback reports: monthly summaries of GPU spend by team, broken down by workload. Teams that see their GPU costs tend to reduce them.

To enforce GPU budgets at the namespace level, apply a ResourceQuota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: team-ml
spec:
  hard:
    requests.nvidia.com/gpu: "8"
    limits.nvidia.com/gpu: "8"

This prevents the team-ml namespace from requesting more than 8 GPUs regardless of what the DCGM data shows. Pair quotas with your showback reports: when a team consistently uses 60% of their quota, you have data to justify a quota increase or a consolidation conversation.

Conclusion

GPU cost monitoring in Kubernetes requires three things working together: a metrics pipeline (DCGM), attribution logic (namespace and labels), and automated action on idle signals. Without all three, you have dashboards but not control.

At 5% average GPU utilization across Kubernetes clusters, the math is straightforward. Most clusters carry substantial idle GPU cost. Therefore, the first engineering investment is visibility: deploy dcgm-exporter, build the idle detection queries, and connect utilization data to team-level attribution.

From there, the path to savings is direct: alert on idle, right-size requests, automate scale-down, and set team budgets. For deeper coverage of the optimization layer that sits above monitoring, see Kubernetes GPU Optimization.

Frequently Asked Questions