KV Cache Explained 2026: Why Long Chats Eat GPU Memory

Why GPU Memory Runs Out Before Compute Does

Model weights are loaded once and stay a fixed size for the life of the deployment. The KV cache is different: it is created fresh for every conversation, grows with every token typed and every token generated, and multiplies again for every user being served at the same time. That is why a GPU can have plenty of spare compute and still refuse a new chat request — it has simply run out of memory to hold everyone's context.

How fast that memory fills up is decided by the model's architecture — its number of layers, attention heads and head dimension — and by the numeric precision used to store the cache, not just by which GPU was bought. Two models with identical weight sizes can need very different amounts of serving memory once real conversations start, which is the point UK buyers most often miss when comparing supplier quotes on GPU specification alone.

What the KV Cache Actually Stores

Attention works by letting each new token compare itself against every earlier token in the conversation, using a "key" and "value" vector produced for that earlier token at every layer and every attention head. Recomputing those key and value vectors from scratch for the whole conversation every time a new token is generated would be enormously wasteful, so inference servers cache them instead — hence "KV cache". Understanding this matters when reasoning about LLM tokens and context windows, because every additional token retained in context is another set of vectors sitting in memory.

This is also why the cache sits alongside the model weights in GPU VRAM: it is not optional working space that can be discarded between requests if the conversation is expected to continue, and it does not shrink back down until the session ends or the tokens fall out of the context window.

The Formula That Actually Sets Your VRAM Bill

The general KV-cache memory formula, as published in current technical guides, is 2 × L × H_kv × d_head × N × B × bytes, where L is the number of transformer layers, H_kv is the number of KV heads left after any grouped-query attention reduction, d_head is the dimension of each head, N is sequence length in tokens, B is batch size (concurrent sequences), and bytes is the storage width per element. NVIDIA's own guidance expresses the same relationship per token as 2 × num_layers × (num_heads × dim_head) × precision in bytes.

The leading 2 in both versions exists because every token, at every layer, at every head, produces both a Key vector and a Value vector that must be stored — not one.

• •L — number of transformer layers in the model
• •H_kv — KV heads remaining after grouped-query attention
• •d_head — dimension of each attention head
• •N — sequence length (tokens in context)
• •B — batch size (concurrent sequences being served)
• •bytes — storage width per element (precision)

Worked Example: One Token to a Full 128K Conversation

Plugging real numbers into that formula shows why long chats bite so hard. A published engineering example uses a 64-layer model with 8 KV heads, 128-dimension heads and FP16 precision (2 bytes per element) and works out to roughly 256 KB of KV cache for a single additional token.

Scale that same model out to a 128,000-token context and a single conversation needs roughly 32 GB of KV cache on its own — before the model's weights are even counted, and before a second user joins. Because the formula is linear in batch size, a second concurrent 128K-context chat on that same model would roughly double that requirement, and a third would roughly triple it, which is precisely why concurrent long-context sessions — not raw parameter count — are the workload that empties VRAM fastest.

Precision is the other lever in that same equation. If that 128K-token cache were stored in FP8 instead of FP16, halving the bytes per element from 2 down to 1 would roughly halve the footprint implied by the formula — illustrating why the cache's storage format is a direct capacity lever, independent of buying a bigger GPU. For teams sizing hardware against this, it is worth cross-checking against how much VRAM an LLM needs before committing to a GPU order.

Why the Model's Architecture Matters as Much as the GPU

Older, smaller models never ran into this problem because their numbers were tiny on every axis. GPT-2 small, for example, has 12 transformer layers, 12 attention heads with no KV-head reduction, a head dimension of 64, and a context window of just 1,024 tokens — a combination that keeps its KV cache trivial by comparison.

Modern open models take a different approach. Llama 3.1 8B uses 32 query heads but only 8 KV heads through grouped-query attention. Because H_kv sits directly in the multiplication, cutting the KV-head count from 32 to 8 shrinks the cache substantially compared with a model that stored one KV head per query head — and it does this at the architecture level, before a single GPU is chosen.

The buyer takeaway: two models with similar weight sizes can require very different amounts of serving memory purely because of how many KV heads each one keeps, which is exactly the kind of specification worth checking before comparing GPU price sheets.

Quantising the Cache: FP16, FP8 and NVFP4

NVIDIA's own long-context inference guidance describes 16-bit as the native KV-cache precision, and separately describes quantising that cache down to 4-bit (NVFP4) as a way to stretch context length and batch size further on the same hardware. NVIDIA Research reports that NVFP4 KV cache can cut footprint by up to about 50% versus FP8, with under 1% accuracy loss — positioning cache quantisation as an accuracy-preserving way to serve much longer contexts on the same card.

These techniques operate at different levels of the stack, which matters when planning a deployment: grouped-query attention is baked into the model's architecture and cannot be switched on afterwards, whereas FP8 or NVFP4 caching are typically inference-time settings that can sometimes be applied to a model regardless of how it was trained. Production inference stacks also commonly manage cache allocation more efficiently and reuse cached prefixes across requests that share the same system prompt, both of which reduce effective memory pressure without touching precision at all — though how much either saves depends on the specific workload mix rather than a single published figure.

The UK Buyer's Checklist Before Comparing GPU Prices

Because KV-cache growth is driven by model architecture and precision as much as by the GPU itself, UK teams evaluating supplier quotes should compare model specifications first: maximum context length, KV-head count relative to query heads, and whether the serving platform supports lower-precision or quantised caching — before comparing GBP price per card, because these specs decide how many concurrent chats one GPU can actually hold.

For regulated UK workloads, it helps to frame KV-cache growth as a capacity-planning and data-governance question rather than a pure performance metric: more cache pressure per user means either fewer users per instance or more instances, which changes both total spend and how many separate systems are handling conversation data. That framing is worth applying before weighing a self-hosting LLM vs cloud GPU cost decision against a managed UK service.

• •Check maximum supported context length, not just parameter count
• •Compare KV-head count to query-head count (grouped-query attention ratio)
• •Confirm whether the platform supports FP8/NVFP4 or only native FP16/BF16 cache
• •Ask whether the serving stack reuses cached prefixes across requests

Conclusion: Cache Is the Real Concurrency Limit

The KV cache, not the model weights, is usually what decides how many simultaneous long chats a GPU can serve — and it is governed by a short, legible list of variables: layers, KV heads, head dimension, precision, context length and batch size, all multiplying together.

Because the cache scales independently of the weights and can, in principle, be reused rather than recomputed for shared context, some teams are starting to size and manage it as a distinct resource rather than an incidental by-product of serving. As 2026 models push further into very long contexts, that shift in how the cache is treated is worth watching as closely as the next GPU generation.

Sources

Every figure in this article traces to the sources below.

• •Lyceum — general KV-cache memory formula and FP16/FP8 byte widths
• •Spheron — 2026 guide confirming KV-cache scaling variables
• •Acing AI — worked example of per-token and 128K-context KV cache size
• •NVIDIA — native 16-bit KV cache and NVFP4 quantisation for long context
• •NVIDIA Research — NVFP4 KV cache footprint and accuracy loss vs FP8
• •Modular — GPT-2 small and Llama 3.1 8B architecture figures