The question sounds simple, but a trustworthy answer comes from one sum, not a rule of thumb. You add the model's weights, the KV cache that grows with context and concurrency, and a slice of overhead, then divide by the usable VRAM on a card. Precision and user load move the total sharply, and Mixture-of-Experts models with latent attention behave differently again. The calculator below runs the maths live for exact GPU counts, VRAM, power, cooling and cost — the copy here explains what it is doing.
| Model | Params | GPUs (H100) | VRAM | From |
|---|---|---|---|---|
| Llama 3.1 8B | 8.03B | 1× | 48.9 GiB | £1,720/mo |
| Mistral Small 3 (24B) | 23.6B | 2× | 86.6 GiB | £2,279/mo |
| Gemma 2 27B | 27.2B | 4× | 146.5 GiB | £3,397/mo |
| Llama 3.3 70B | 70.6B | 4× | 216.7 GiB | £3,397/mo |
| Qwen2.5 72B | 72.7B | 4× | 220.7 GiB | £3,397/mo |
| Mixtral 8x7B | 46.7B | 2× | 122.4 GiB | £2,279/mo |
| DeepSeek-R1 (671B MoE) | 671B | 24× | 1293.3 GiB | £16,512/mo |
All figures on this page are indicative estimates for planning only and are subject to change; hardware and throughput vary with configuration and tuning, and any monthly finance figure is subject to credit approval and is not a quotation.
Start with the sum: weights + KV cache + overhead
Every GPU count starts with one sum: model weights plus KV cache plus overhead. Weights are the fixed cost — parameters multiplied by the bytes each one occupies. The KV cache is the variable cost, growing with every token of context and every simultaneous request you serve. Overhead covers the CUDA context, activations and framework slack. Add the three, divide by a GPU's usable VRAM, and you have a floor for the number of cards.
Why a sparse MoE model still fills every card
The weights term is where this workload gets interesting. Its published parameter count is enormous, but it is a Mixture-of-Experts model: only a small fraction of experts fire on any given token. Every expert still has to be resident in VRAM, so the weights term is set by the total parameter count, not the active slice. That is why an MoE model of this scale needs a full multi-GPU node even though each token touches only a sliver of the network.
Latent attention keeps the KV cache small
KV cache is normally the term that explodes with long context, but this architecture uses Multi-head Latent Attention. Instead of caching keys and values for every attention head, it stores a single compressed latent per token per layer. The saving is dramatic against its large head count, so its cache stays modest even as context runs into six figures. Naive calculators that count query heads overstate this badly; the tool below caches the latent correctly.
FAQs
If only a few experts activate per token, why does it need so much VRAM?
Weights are set by the total parameter count because every expert must sit in VRAM even though only a handful activate per token. The KV cache, by contrast, tracks active compute and stays small thanks to Multi-head Latent Attention. So memory is dominated by the full weight set, which is what forces a multi-GPU node regardless of the low active-parameter figure.
Does the long context window blow up the memory?
Not the way you would expect — a compressed latent cache means context length costs far less memory here than on a comparable attention design. You can push toward its very large context ceiling without the KV cache dominating VRAM. It still grows with concurrency, so many long-context sessions at once add up. Set your real context and user count in the calculator to see the true figure.
Can I run it on a couple of cheaper GPUs without NVLink?
Not comfortably. This model's full weight set spans several high-VRAM cards that must act as one, which needs NVLink and NVSwitch bandwidth between them. Cards without NVLink, or with too little VRAM, either cannot hold it or bottleneck on cross-GPU traffic. The calculator steers you to a suitable NVLink node and shows the resulting count, power and cost.