GPU Utilisation Monitoring: Fix It Before Buying More GPUs

Why your GPU estate is a black box — and why that's costing you money

Every GPU vendor's pitch in 2026 assumes your existing fleet is maxed out and the only lever left is more silicon. The evidence says otherwise. ValueAddVC's June 2026 survey of 573 technical leaders at companies with 100+ employees found that 86% of enterprises running their own GPU infrastructure reported utilisation of 50% or less; VentureBeat's separate write-up of the same underlying survey put the figure at about 83%. Either way, the majority of self-managed GPU estates are running at half capacity or worse — and that's before you look at cluster-level telemetry.

That telemetry is starker still. CAST AI's 2026 analysis of roughly 23,000 production Kubernetes clusters across AWS, Azure and Google Cloud, reported by both The Register and TechRadar, found average GPU utilisation of just 5%. That's not the same measurement as the survey — one is operators estimating fleet-wide usage, the other is fine-grained cluster telemetry — but both point the same direction: enterprises are paying for far more GPU than they're running.

For UK buyers the stakes are higher because the hardware itself carries a local premium. A UK-listed NVIDIA H100 PCIe was reported at £32,050 in June 2026, including VAT and import duties. Buying another one of those to solve what is very likely a scheduling, batching or data-pipeline problem is an expensive way to end up exactly where you started.

The essential GPU metrics: what to watch and what the values actually mean

Most utilisation debates get lost because teams look at one number in isolation. Microsoft's Azure documentation confirms that NVIDIA's DCGM exporter collects and exports a coherent set of GPU metrics — utilisation, memory usage and power consumption together — precisely so they can be read as a set, not individually.

• •DCGM_FI_DEV_GPU_UTIL — NVIDIA's own cluster-efficiency work uses this counter as the primary indicator of GPU activity. Sustained near-zero readings during business hours, not brief dips, are the clearest evidence of stranded capacity.
• •Memory usage and power draw, read alongside utilisation — high power draw with low utilisation usually points to a memory-bound or data-starved job rather than genuinely idle hardware; both are exported together in the DCGM/Azure Monitor path.
• •Health signals (ECC error trends, thermal thresholds) — if these climb while utilisation stays flat or low, treat it as a hardware problem to isolate, not a scheduling gap to fix with more capacity.
• •Idle duration — NVIDIA defined a workload as idle when it detected a full hour of continuous GPU inactivity in its own cluster-efficiency analysis. That's a defensible, tested threshold for triggering an alert rather than reacting to normal per-minute noise.

Step by step: building the DCGM, Prometheus and Grafana stack

You don't need a commercial platform to see whether your GPUs are actually working. NVIDIA's own November 2025 engineering blog describes aligning real-time DCGM telemetry with Slurm job metadata to build a usable utilisation pipeline — the same pattern applies whether your scheduler is Slurm or Kubernetes. Microsoft's Azure Monitor guidance confirms this is a standard, supported path: DCGM exporter metrics can be queried directly through Azure Managed Prometheus and Azure Managed Grafana, so cloud-native teams don't have to run the observability layer themselves.

• •Deploy the DCGM exporter on every GPU node (as a container or daemonset on Kubernetes) so each host exposes its DCGM metric set — utilisation, memory, power — on a scrapeable endpoint.
• •Point Prometheus (self-managed, or Azure Managed Prometheus if you're on Azure) at that endpoint across the fleet. This is the exact integration Microsoft documents for DCGM.
• •Connect Prometheus to Grafana, or Azure Managed Grafana, and build a first dashboard around DCGM_FI_DEV_GPU_UTIL per node and per job, with power and memory overlaid and an idle panel built on the one-hour continuous-inactivity rule NVIDIA used in its own analysis.
• •If you run Slurm, replicate NVIDIA's approach and join DCGM telemetry to job metadata so utilisation is attributable to a user, team or workload — not just an anonymous GPU ID. This attribution step is exactly what separates a credible utilisation report from the kind of fleet-wide estimate behind the 86% survey figure.
• •Add queue-time and job-wait metrics next to utilisation before any purchase conversation. A GPU running at 90% with nothing queued behind it is not evidence of a capacity shortfall.

Troubleshooting and anomaly detection: the pitfalls that waste engineering time

The most common mistake is comparing figures with different scopes as if they contradict each other. A fleet-wide self-reported estimate (the 86%/83% survey figures) and a per-cluster telemetry average (CAST AI's 5%) are measuring different things at different granularity — treat them as two separate warnings, not one number.

The second mistake is chasing 100% as a target. Anyscale's January 2026 analysis, cited by VentureBeat, argued that modern containerised AI workloads can sit below 50% GPU utilisation even when the fleet is correctly sized, simply because of how containers and job structure work. Alerting on every dip below an arbitrary high-water mark generates fatigue and gets ignored; alerting on NVIDIA's one-hour continuous-inactivity definition is a far more defensible anomaly-detection threshold.

Two concrete diagnostic scenarios worth building into dashboards: a GPU showing high power draw and temperature with a flat, low utilisation curve usually indicates a driver or throttling issue, not real workload demand — check power alongside utilisation before assuming the job itself is heavy. A GPU with climbing error counts and falling utilisation is a hardware health problem; pull it from the scheduling pool rather than adding a new node to compensate.

Bare-metal, virtualised and Kubernetes: monitoring looks different in each

On bare metal, the DCGM exporter reads the physical card directly — the cleanest possible signal, but with no built-in concept of multiple tenants sharing a GPU.

In virtualised environments (VDI, GPU passthrough or vGPU), a hypervisor layer sits between DCGM and the physical card, which can undercount contention; per-VM attribution needs the hypervisor's own counters read alongside DCGM, not DCGM alone.

On Kubernetes, DCGM exporter runs as a daemonset and slots into the Azure Managed Prometheus/Grafana path Microsoft documents. It's also exactly the environment CAST AI's 23,000-cluster analysis was drawn from, and its 5% average utilisation figure is a direct warning about container-scheduling defaults: reserving a whole GPU per pod regardless of actual use is a routine way to strand capacity at scale.

Mixed NVIDIA/AMD estates add a further wrinkle. DCGM is NVIDIA's own toolkit, so a genuinely unified view across vendors means running a second, vendor-appropriate telemetry pipeline for non-NVIDIA cards and standardising on Prometheus and Grafana as the common backend, rather than expecting a single exporter to cover every GPU in the estate.

The compliance angle: power telemetry isn't optional data

Power consumption is one of the three headline metrics the DCGM exporter collects alongside utilisation and memory — and that's not incidental. GPU-level power draw, logged over time, is the raw input any UK operator needs to produce a credible energy-efficiency or data-centre sustainability figure. If you can't currently produce a power-per-GPU trend from your own estate, you also can't produce a credible efficiency case for expanding it — a gap that increasingly matters to both internal ESG reporting and to boards asking why infrastructure is running warm rather than hot.

Turning the data into savings: what idle GPUs actually cost in the UK market

If you're renting rather than owning, measurement discipline pays back even faster because the rental market is so widely dispersed. Servnet UK's mid-2026 GPU rental index recorded AWS H100 on-demand pricing at about $6.88 per GPU-hour in March 2026 and Azure ND H100 v5 instances at about $12.29 per GPU-hour, also in March 2026. CoreWeave's European on-demand rate was $49.24 per hour for a full 8-GPU HGX H100 node in July 2026 — equivalent to $6.16 per GPU-hour — while Spheron listed H100 on-demand pricing at $2.01 per hour in May 2026. Across the broader tracked market, on-demand rates ran as high as $14.90 per GPU-hour, with spot pricing as low as $0.36 per GPU-hour in July 2026. None of that spread matters if the GPU you're already renting is sitting at anything close to CAST AI's cluster-wide average of 5% — you're paying premium on-demand rates for silicon that's doing nothing.

On the ownership side, a UK-listed H100 PCIe at £32,050 including VAT and import duties (June 2026) is an expensive way to paper over a scheduling problem. Run the DCGM/Prometheus numbers for a full reporting cycle first, weigh them against the self-hosting LLM vs cloud GPU cost maths, and — only if a genuine gap remains — model the purchase against financing your AI build options before committing capex.

Decision framework: which monitoring approach, and when to actually buy

Match the stack to your scale and skills, not to whatever a vendor is pitching this quarter. A single-node or small-team setup gets everything it needs from a self-hosted DCGM exporter, Prometheus and Grafana build — low cost, but it needs someone comfortable running Linux infrastructure. A cloud-native or Kubernetes-heavy shop should default to the managed path Microsoft documents — Azure Managed Prometheus and Managed Grafana reading DCGM metrics directly — to avoid running the observability layer as a second job. A large, multi-vendor estate needs the extra step of joining DCGM (or the equivalent non-NVIDIA telemetry) to Slurm or Kubernetes job metadata so utilisation is attributable to a team, not just a device ID.

The verdict for UK IT leaders is straightforward: the burden of proof for a new GPU purchase or reserved cloud commitment should be a utilisation and queue-time report, not a growth forecast. Only once that report shows genuine, sustained contention should you size the real gap with the AI GPU calculator — and buy for it, not around it.

Sources

Every figure in this article traces to the sources below.

• •ValueAddVC — June 2026 survey of 573 technical leaders on enterprise GPU utilisation
• •VentureBeat — write-up of the same survey (83% figure)
• •VentureBeat — Anyscale analysis on containerised workloads and sub-50% utilisation
• •TechNewsWorld/The Register — CAST AI analysis of ~23,000 Kubernetes clusters
• •TechRadar Pro — coverage of CAST AI's 5% utilisation finding
• •Microsoft — Azure Monitor DCGM exporter and Managed Prometheus/Grafana integration
• •NVIDIA — DCGM/Slurm telemetry pipeline and idle-workload definition
• •Servnet UK — UK GPU cloud rental price index 2026
• •Servnet UK — AI workload repatriation business case (UK H100 pricing)