As agentic AI pushes context windows and multi-turn reasoning further, KV cache growth is overwhelming GPU and system memory. Huawei's OceanStor M900 Context Memory Storage, unveiled at HUAWEI CONNECT 2026, proposes a dedicated memory tier to keep accelerators fed — a signal UK buyers should optimise your infrastructure for AI inference rather than just add GPUs.
View the data behind this chart
| Layer | Detail |
|---|---|
| Accelerator memory (HBM) | Fastest and most expensive, smallest capacity |
| System memory (DRAM) | Fast access, costly, capacity constrained |
| Context Memory Storage | Shared KV cache pool, 60-microsecond access via UnifiedBus |
| Traditional storage | High capacity but CPU-mediated, higher latency |
Why KV cache is becoming the real constraint
Every inference request against a large language model generates a key-value cache — a running record of previously processed context that lets the model avoid recomputing it. The brief underpinning this story is blunt about the problem: as organisations move from simple prompts to long conversations, complex reasoning and autonomous agents, context volumes balloon and so does the KV cache that supports them. GPU memory and system memory can hold this cache, but both are expensive and limited in capacity, while conventional storage, though roomier, routes data through CPU-mediated paths that introduce latency and leave accelerators idle while they wait.
For UK buyers, this matters because it reframes a purchasing conversation that has, until now, been almost entirely about accelerator count. If the bottleneck sits in memory and data movement rather than raw compute, then adding more GPUs to a cluster without addressing how context is stored and retrieved risks buying expensive hardware that spends much of its time waiting. Readers unfamiliar with the mechanics involved should understand the KV cache in LLM inference before sizing the next procurement cycle.
What Huawei announced at HUAWEI CONNECT 2026
At HUAWEI CONNECT 2026 in Shanghai, Huawei introduced OceanStor M900 Context Memory Storage, built to support AI inference across hyperscale datacentres and its Ascend SuperPoD infrastructure. The platform adds what Huawei calls a dedicated Context Memory Storage layer, extending the KV cache beyond accelerator and system memory into a shared, multi-tier storage pool accessible across a cluster via Huawei's UnifiedBus high-speed interconnect.
According to the brief, Huawei claims the platform can deliver up to 64PB of capacity and substantially more context available per NPU. Its ASU disk integrates CPU, networking and storage controller functions, and combined with native KV semantics this lets NPUs access stored cache data in a single hop — bypassing the protocol conversion and CPU forwarding that typically slow conventional storage access down. Huawei states this approach reduces access latency to 60 microseconds, with claimed effects of doubling token throughput and halving time to first token in typical AI coding workloads.
The shift from GPU expansion to storage-accelerated inference
The analytical point for UK infrastructure buyers is that this announcement treats storage, networking and compute as interdependent rather than separate line items. Multi-GPU clusters built purely around accelerator density can still bottleneck badly if the path between storage and compute is slow, particularly as agentic workloads run longer, multi-step reasoning chains that keep context alive for extended periods. A cluster with abundant GPUs but a conventional storage tier may simply push the wait time from the accelerator queue into the data path instead.
This is why buyers scaling multi-GPU deployments in 2026 need to budget time and spend for storage-accelerated inference architecture, not just GPU counts. Before finalising a cluster spec, it is worth using a tool to calculate your AI GPU requirements alongside a realistic view of expected context length and concurrency, since those numbers determine how much pressure will land on whatever memory tier sits behind the accelerators.
Cost, endurance and the total-ownership picture
Huawei's brief also foregrounds durability economics. Its KV-aware adaptive storage technology manages data placement according to cache lifecycles, with claimed endurance of up to 24 drive writes per day — stated as 16 times that of conventional SSDs. Huawei argues this reduces replacement and maintenance costs over time, which is a relevant consideration for UK buyers weighing the lifetime cost of high-throughput inference storage against the churn rate of components under constant cache writes and rewrites.
That endurance claim is worth scrutinising against your own workload profile rather than taken at face value, since drive-writes-per-day figures are heavily dependent on access patterns. But the direction of travel — storage designed explicitly around inference cache lifecycles rather than general-purpose throughput — is a useful framework when evaluating vendor proposals, and buyers should explore unified storage for AI analytics options that are built with similar cache-lifecycle awareness in mind.
View the data behind this chart
| Conventional SSD | Huawei ASU Disk | |
|---|---|---|
| Drive writes per day | DWPD1.5 | DWPD24 |
What this means for UK procurement decisions now
Huawei frames the wider significance of OceanStor M900 as a change in how memory, networking and compute are positioned relative to one another — not as separate supporting components but as increasingly interconnected elements of inference performance. For organisations in the UK planning or expanding AI infrastructure, the practical takeaway is to treat memory tiering as a first-class design decision alongside accelerator selection, power provisioning and networking.
Teams building their first production AI cluster should factor storage-tier architecture into the design from day one rather than retrofitting it once GPU utilisation figures expose the bottleneck. It's worth reviewing guidance to build your first on-prem AI cluster with this in mind, and pairing it with work to address AI power bottlenecks, since power, memory and storage constraints tend to surface together as agentic workloads scale.