AI and Token Cost Management on Kubernetes: Attributing Inference Spend to Teams

Key Takeaways

• Token cost attribution is a join problem: two billing systems (cloud node-hours and model API tokens) share no common key by default. Closing the gap requires propagating a workload label all the way through to the inference billing record.
• Average GPU utilization sits at 5% across major clouds (2% on AKS, 5% on EKS, 6% on GKE). Most of the hourly bill is idle capacity, making attribution the prerequisite for any cost reduction work.
• Self-hosted and API-based inference are two different attribution problems. Self-hosted requires joining vLLM Prometheus metrics to node cost. API-based requires team identifiers attached at call time via metadata objects or per-team API keys.
• The five-step fix: define a label convention, enforce it at admission with Kyverno, expose the team label via the Downward API, route calls through a gateway like LiteLLM, then join GPU cost to token volume with a PromQL query.
• FOCUS 1.4 (ratified June 2026) normalizes billing records across clouds and model APIs but does not yet cover the Kubernetes namespace-to-token join. FOCUS 1.5 is expected to address this gap.
• Start with showback before chargeback: publish spend reports by team label and use LiteLLM budget caps for near-real-time control. Monthly chargeback cycles are too slow for AI spend patterns.

AI inference costs are now a real budget line, and two Cast AI posts already make that case at length. This post skips that argument. Instead, it covers the attribution mechanics: how you get your Kubernetes cost tooling and your model spend to agree on who owns what. If you are working on Kubernetes FinOps more broadly, inference attribution is the next layer to solve.

Why token cost does not show up in your Kubernetes cost tool

Two billing systems, no shared key

Your Kubernetes cost tool, whether OpenCost, Kubecost, or a cloud-native equivalent, reads from two sources: the cloud provider billing API for node costs, and the Kubernetes API for workload metadata. It produces allocation by namespace, label, and workload. That is its model. It has no concept of tokens.

Your model API produces a billing record keyed by API key, project, or organization. OpenAI bills by project and API key. Anthropic bills by workspace. Neither record carries a Kubernetes namespace, a pod label, or any signal from the cluster. The two systems share no join key by default.

The result is a gap that is structural, not accidental. No amount of tagging your cloud nodes fixes it, because the token cost lives in a different billing system entirely. The only way to close the gap is to propagate a shared identifier from the workload side all the way through to the inference request, then join on that identifier at report time.

Self-hosted inference versus API-based: two different attribution problems

The gap looks different depending on where the model runs. Self-hosted inference means you own the GPU node. The cost signal is node-hours, and the attribution challenge is determining which workload on that node consumed how much compute. API-based inference means you pay a provider per token, and the attribution challenge is tracing which team or service made which API calls.

These are two different problems requiring different instrumentation, different join keys, and different reporting. The table below shows the contrast.

If your platform runs both self-hosted and API-based inference – which is common as teams migrate high-volume workloads in-house while keeping experimental models on managed APIs – you need both attribution paths running simultaneously.

What you are actually trying to attribute

GPU node hours for self-hosted models

To attribute GPU node cost to teams, you need two things: the hourly node cost from cloud billing, and a metric that ties inference requests to workload labels. When you run vLLM, TGI, or a similar serving stack on GPU nodes, the cost is denominated in node-hours. AWS H100 marketplace pricing runs at approximately $6.88 per hour as of mid-2026 (AWS EC2 p5.48xlarge H100 on-demand at ~$12–13/hr per card). Azure prices equivalent H100 hardware (ND H100 v5-series) at around $12.29 per hour in comparable regions as of mid-2026. Those costs accrue whether the model is processing requests or idling. Average GPU utilization sits at 2% on AKS, 5% on EKS, and 6% on GKE (Cast AI 2026 State of Kubernetes Optimization Report), with a cluster-wide average of 5%. Most of that hourly bill represents idle capacity.

vLLM exposes the right Prometheus metrics for the usage side of the join: vllm:prompt_tokens_total, vllm:generation_tokens_total, and vllm:kv_cache_usage_perc (renamed from vllm:gpu_cache_usage_perc in an earlier v0.x release — check your vLLM release notes for the exact version boundary before assuming which metric name your deployment exports). OpenCost 1.121.0 (released July 2026, per the OpenCost changelog) is the first Kubernetes-native inference cost tracking release. It integrates vLLM metrics directly with its GPU allocation engine, producing four cost views: allocation-based, usage-based, per-model, and per-token.

Token spend for API-based models

To attribute spend from external model APIs, you need a team identifier attached at call time – before the billing record is written, not after. The cost unit is tokens, and the identifier must travel with each request consistently across every service that calls the API. OpenAI’s billing API lets you attach a metadata object to each request, and project-level API keys let you segment spend by team before the billing record is even written. Anthropic provides workspace IDs accessible via the Admin API billing export. Both approaches let you attach a team identifier at call time, but they require discipline: every service that calls the API must include the identifier consistently.

The failure mode is shared API keys. When multiple services share one key, attribution collapses to a single billing line for the entire organization. You can only recover attribution by parsing application logs, which is fragile and often incomplete.

The shared overhead: gateways, vector stores, caches

Beyond node costs and token spend, several shared components sit between the application and the model: LLM gateways, vector databases, and semantic caches. These run as Kubernetes workloads, so their costs are attributable through standard namespace and label allocation. The complexity is that their cost is joint: a shared vector store serves multiple teams, and splitting that cost requires a usage signal, not just a label.

Gateways like LiteLLM record per-request metadata and produce per-team usage summaries. Vector stores like Qdrant and Weaviate emit request metrics you can aggregate by calling service. These signals exist; the work is collecting and joining them.

A less obvious shared cost is the KV cache prefix hit. When multiple teams share a vLLM deployment, the first request to load a given prompt prefix pays the full context-loading cost; subsequent requests from other teams that reuse the same prefix hit the cache and pay nearly zero compute. The attribution policy decision — socialize this cost across all beneficiaries, or bill it to the first caller — has no universally correct answer. Billing the first caller creates a perverse incentive to delay requests. Socializing the cost is fairer over time but requires prefix-cache hit-rate tracking per team that most platforms do not have on day one. A practical middle ground is to track cache hit rates over a rolling window and apply a discount to high-hit-rate callers, funded by a pro-rated surcharge on cache misses.

How to build the join

The join requires five steps in order. Complete steps 1–3 before step 4, or your LiteLLM virtual key mapping will have no namespace context and spend will be recorded against an unidentified key.

1. Define and apply a label convention to every inference workload.
2. Enforce labels at admission with a Kyverno ClusterPolicy.
3. Expose the team label as an environment variable via the Downward API.
4. Create LiteLLM virtual keys per team and propagate them through your gateway.
5. Build the PromQL query that joins token volume to node cost.

Steps 1–3: Labelling workloads so the key survives to the inference request

Attribution starts with a label convention enforced at admission. The label set that works for inference attribution is: app.kubernetes.io/team, app.kubernetes.io/cost-center, environment, and model-name. Enforce these at admission using Kyverno or OPA/Gatekeeper so that no workload reaches the scheduler without them. A missing label at deploy time is far easier to catch than a missing attribution record six weeks later when finance asks who owns the spike.

A namespace and label convention that works

The following manifest shows a label convention for a team-scoped inference workload. The labels are consistent across the namespace, the deployment, and the pod template, so every cost tool that reads pod metadata produces the same attribution key. The TEAM_LABEL environment variable (step 3) is injected via the Downward API directly from the pod’s own label, keeping the label and the runtime value in sync without manual duplication.

# Namespace with team and cost-center labels
apiVersion: v1
kind: Namespace
metadata:
  name: ml-team-search
  labels:
    app.kubernetes.io/team: search
    app.kubernetes.io/cost-center: cc-1042
    environment: production
---
# Deployment inheriting the namespace convention
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-server
  namespace: ml-team-search
  labels:
    app.kubernetes.io/team: search
    app.kubernetes.io/cost-center: cc-1042
    app.kubernetes.io/component: inference-server
    model-name: meta-llama-3-8b-instruct
    environment: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama3-server
  template:
    metadata:
      labels:
        app.kubernetes.io/team: search
        app.kubernetes.io/cost-center: cc-1042
        model-name: meta-llama-3-8b-instruct
        environment: production
    spec:
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - "--model"
          - "$(MODEL_NAME)"
          - "--port"
          - "8000"
        resources:
          limits:
            nvidia.com/gpu: "1"
          requests:
            nvidia.com/gpu: "1"
        env:
        - name: TEAM_LABEL
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['app.kubernetes.io/team']

The MODEL_NAME value should match the model identifier served by vLLM (e.g., meta-llama/Llama-3.1-8B-Instruct). GPU resource limits are required. Without them, the pod schedules on a CPU node and the vLLM process fails immediately.

Enforce this schema with a Kyverno ClusterPolicy that mutates missing labels at admission and validates required fields. Pods that arrive without app.kubernetes.io/team should be rejected in production namespaces. The policy is your contract that attribution data will be clean at collection time.

Step 4: Propagating the key through a gateway or proxy

With labels in place and the team identifier available as TEAM_LABEL inside each pod, the next problem is carrying that identifier through to the inference request – especially for API-based models where the request leaves the cluster entirely. This is where a gateway like LiteLLM becomes the critical path: it translates the Kubernetes identity into a spend record that your billing layer can join on.

LiteLLM attributes spend through its virtual key model: each API key is created in the proxy and assigned to a team in the Org > Team > User > Key hierarchy before any inference request is made. When a request arrives with a virtual key in the Authorization header, LiteLLM looks up the associated team and writes a spend record. Budget enforcement fires at the team level. To attribute spend, generate a virtual key per team, distribute it to that team’s services, and use LiteLLM’s /team/info endpoint to query spend by team.

The gateway also forwards the team header to vLLM, so token metrics carry the team label through to Prometheus, for self-hosted models. For OpenAI calls, include the team identifier in the metadata object on each request and use per-team project API keys. For Anthropic, assign workspace IDs per team and use the Admin API billing export to retrieve spend by workspace. Both approaches require that every service calling the API follows the same convention consistently.

Step 5: Joining GPU node cost to workload

Once labels are in place and gateway attribution is running, the final step is joining node cost to workload. Standard Kubernetes cost allocation tools handle this for CPU and memory workloads. For GPU inference, the join needs to combine node cost with vLLM token metrics, using token volume as the usage weight.

The token_fraction_by_team metric is the right attribution unit here – not a pod resource share – because it weights cost by actual inference workload rather than by what was requested. A pod that requests a full GPU but processes 2% of total tokens should carry 2% of GPU cost, not 100%. A pod-level resource share would assign the full GPU cost to that pod even if it spent most of the billing period idle. Token fraction captures actual usage; resource reservation does not. That distinction matters most in shared-GPU environments where multiple teams submit requests to the same serving instance.

Before running this query: vLLM pod labels do not propagate to Prometheus automatically – you need kube-state-metrics configured with --metric-labels-allowlist=pods=[app.kubernetes.io/team,app.kubernetes.io/cost-center] to export team labels as metric labels on kube_pod_labels. Additionally, this query assumes your namespace name exactly matches your team label value (e.g., namespace search = team search). If your namespace naming convention differs from team naming, replace the label_replace step with a join on app_kubernetes_io_team exported via kube-state-metrics’ --metric-labels-allowlist flag.

# Step 5: token fraction per team for the last hour
# vLLM exposes prompt_tokens per model and pod
token_fraction_by_team = sum by (team) (
  label_replace(
    increase(vllm:prompt_tokens_total[1h]),
    "team", "$1", "namespace", "(.*)"
  )
) / ignoring(team) group_left sum(increase(vllm:prompt_tokens_total[1h]))

# Attributed cost = fraction x (GPU count x on-demand rate x hours)
# Replace GPU_COUNT and HOURLY_RATE with your cluster values, or pull from billing API
# Example for a single H100 node at $6.88/hr (marketplace) over 1 hour:
attributed_cost = token_fraction_by_team * GPU_COUNT * HOURLY_RATE * 1

In practice, the node cost figure comes from your cloud billing export rather than a hardcoded constant. Record it as a Prometheus recording rule fed from a billing scraper, or pull it in at query time from a BI tool that has already ingested the cost export. For SQL-based reporting, the logic is the same: join the billing export to the Prometheus metric export on the team label, then compute weighted cost shares per reporting period.

The FOCUS specification and where AI cost fits

What FOCUS standardises today

FOCUS 1.4, ratified in June 2026, standardises the Invoice Detail dataset, which normalises billing records from cloud providers, SaaS vendors, and AI API providers into a common schema. A FOCUS-compliant FinOps tool can ingest billing data from AWS, Azure, GCP, and OpenAI using the same field names and report across them without custom mapping. It also standardises the Billing Period dataset for invoice reconciliation.

What it does not yet cover for token and inference spend

FOCUS 1.4 does not standardise the join between Kubernetes namespaces and token spend. There is no specification for how workload context should propagate through the inference stack, and self-hosted inference attribution is explicitly out of scope for the current version. The standard normalises what billing records look like after they are emitted; it does not define how the team identifier gets into the billing record in the first place.

The FOCUS 1.5 specification, currently in development, is expected to introduce Price Sheet data, per-model cost segmentation, and input/output token distinction. Until 1.5 is ratified, practitioners building token attribution today work outside any FinOps standard. Use the FOCUS 1.4 Invoice Detail as a normalisation layer for your API billing export, and build the namespace join on top of it with the custom columns your pipeline adds.

Showback and chargeback for AI teams

Why AI spend breaks a monthly chargeback cycle

Standard infrastructure chargeback runs on a monthly cycle: attribute costs after the billing period closes, reconcile against budgets, and adjust. AI inference spend does not fit that model. Token usage can spike by an order of magnitude within hours, driven by a product launch, a batch job, or an LLM agent loop without a spend ceiling.

Without attribution, AI spend accumulates in a single billing line. Engineering teams cannot answer which model or which team drove last month’s 40% cost increase. That ambiguity blocks any conversation about ROI or budget reallocation.

Showback is a practical starting point. Publishing spend reports to teams by team label without enforcing financial transfers builds cost awareness and reveals data quality problems before they become chargeback disputes. LiteLLM’s budget enforcement at the team and key level handles the near-real-time control problem: set a spend cap per team, per model, or per API key, and the gateway blocks requests when the cap is hit. That operational model fits AI spend patterns far better than a monthly reconciliation cycle ever will.

Reducing the bill once you can see it

Attribution tells you where spend is going. Once you have that visibility, you can reduce it. For a deeper guide to inference optimization, see Cast AI’s post on LLM inference cost optimization and the GPU cost visibility resource.

GPU sharing and time-slicing for self-hosted inference

At 5% average GPU utilization, most inference nodes are mostly idle. GPU time-slicing runs between 1 and 48 virtual replicas on a single physical GPU. MIG (Multi-Instance GPU) partitioning on A100, H100, and H200 hardware creates isolated, fixed-size GPU instances that multiple workloads share safely. GPU sharing reduces cost by 50 to 70% for non-latency-sensitive batch inference workloads, according to workload studies on MIG-enabled clusters (this range varies significantly by workload latency tolerance – time-slicing has higher tail latency overhead under contention, so real-time inference with strict SLA requirements benefits less). Sharing introduces scheduling latency that can breach p99 targets.

The attribution implication: when multiple workloads share a GPU, you need the utilization metric, not just the allocation label, to split cost fairly. The vllm:kv_cache_usage_perc metric (renamed from vllm:gpu_cache_usage_perc in an earlier v0.x release – check your vLLM release notes for the exact version boundary) and per-request token counts give you the usage weight for that split. Without those metrics, shared-GPU cost becomes an average split, which is wrong in both directions for workloads with uneven request patterns.

Rightsizing GPU requests

Most inference workloads request a full GPU when a smaller partition would serve. An 8B parameter model at typical batch sizes fits within a single MIG slice on an H100. Requesting a full node-level GPU for that workload inflates the allocation cost and creates idle capacity that other teams cannot use. Cast AI surfaces per-pod idle GPU utilization directly, so you can see which deployments are over-requesting without manually correlating Prometheus metrics to billing data – so you can see which deployments are over-requesting before they accumulate into a budget problem.

Rightsizing starts with observing actual GPU memory utilization over representative traffic windows, then adjusting requests to match. H200 Capacity Block pricing increased 15% in January 2026, the first GPU price increase in Cast AI’s data (Cast AI 2026 State of Kubernetes Optimization Report). That trend makes rightsizing a more urgent exercise than it was a year ago, especially for teams running development or staging models on production-tier GPU hardware.

Cross-cloud and cross-region capacity

On-demand H100 pricing varies significantly by cloud: AWS H100 marketplace pricing runs at approximately $6.88 per hour as of mid-2026 (AWS EC2 p5.48xlarge H100 on-demand at ~$12–13/hr per card), while Azure H100 (ND H100 v5-series) charges approximately $12.29 per hour in comparable regions as of mid-2026. Spot H100 pricing reaches as low as $1.25 per hour in select regions and availability zones as of mid-2026 – though spot availability and pricing shift frequently, so treat this as a floor to validate against live pricing APIs, not a guaranteed rate. For batch inference workloads that tolerate interruption, spot capacity is the most direct cost reduction available without any model or architecture changes.

Cross-cloud and cross-region placement requires a scheduler that understands GPU availability and pricing across providers. That is harder than single-cloud spot scheduling, but the price differential justifies the complexity for large or long-running inference workloads. Cast AI continuously monitors spot availability and pricing across GPU instance types and regions, moving workloads automatically when a better price or capacity combination becomes available. No manual scheduling change or redeployment required.

Conclusion

The token cost attribution problem is a join problem: two billing systems with no shared key. Solving it requires a label that survives from Kubernetes workload metadata through to the inference billing record, a gateway that carries that label on every request, and a reporting layer that combines node cost with token usage. With average GPU utilization sitting at 5% across major cloud providers (Cast AI 2026 State of Kubernetes Optimization Report), getting attribution right also sets you up for the cost reduction work that follows. Start with Cast AI’s GPU cost visibility to understand where your current GPU spend is going, then book a demo to see how Cast AI’s automation layer that handles rightsizing and scheduling decisions – including OMNI for cross-cloud GPU scheduling – handles attribution and optimization together.

Frequently Asked Questions