Stateful container workloads have decisively outgrown the lab. According to the CNCF’s 2025 Annual Cloud Native Survey (published 20 January 2026), 82% of container users run Kubernetes in production. For infrastructure directors inheriting bare-metal estates, selecting storage solutions is no longer about chasing synthetic IOPS; it is an exercise in failure containment, day-2 operational friction, and staff retention. On-premise deployments cannot rely on hyperscaler APIs to absorb node evictions or unmount timeouts. Instead, platform teams face a harsh trade-off: deploy distributed software-defined storage like Ceph via Rook, take on simpler block-only overlays like Longhorn, or connect enterprise SAN/NAS arrays via CSI drivers. Balancing operational burden against UK compliance requirements and ongoing rack power costs dictates which architecture survives production.
View the data behind this chart
| Rook-Ceph | Longhorn | Array CSI | |
|---|---|---|---|
| Storage Types | Block, File, Object | Block only | Block and File |
| Operational Burden | High (Ceph daemons) | Low (Kubernetes CRDs) | Low (Array offloaded) |
| Failure Domain | Cluster-wide CRUSH | Volume replica sets | Array controller pool |
| Data Engine | Native Ceph OSDs | V2 Data Engine (GA) | Array-native ASIC |
| Min Node Footprint | 3 OSDs, local disks | Multi-node replicas | Node initiator only |
The On-Premise Kubernetes Conundrum: Why Bare Metal Changes Everything
In cloud environments, storage attach and detach operations are handled by underlying hypervisor APIs that abstract device states from physical failures. Running persistent volumes inside an on-premise data centre strips away this safety net. A bare-metal Kubernetes node running stateful applications must directly handle kernel storage mounts, transport-layer network drops, and raw drive resets. When an on-premise node loses network quorum, the Container Storage Interface (CSI) driver must coordinate with the storage control plane to prevent split-brain writes while releasing multi-attach locks.
Infrastructure leads must also reconcile these architectural decisions with physical infrastructure costs. In the UK, commercial data centre power and colocation tariffs—factoring in Power Usage Effectiveness (PUE) overheads alongside volatile wholesale electricity rates—mean that running compute nodes bloated with heavy storage daemons extracts a persistent thermal and electrical toll across multi-rack deployments.
Furthermore, on-premise clusters operate within varied physical connectivity realities. While cloud hyperscalers offer abstracted software-defined networks, internal enterprise data centres often feature uneven top-of-rack switching, heterogeneous NIC capabilities, or multi-site links where inter-node latency spikes above 5ms can quickly stall synchronous volume replication. Your on-premise storage architecture must withstand these transport-layer realities without stalling I/O or corrupting data.
- •Cloud volumes mask underlying hardware faults via control-plane hypervisors; bare-metal CSI drivers require explicit fencing logic.
- •Unplanned node reboots frequently leave PersistentVolumes locked in Terminating states if the CSI driver lacks robust detach timeouts.
- •Storage daemons share memory and CPU cores with application pods, so careful resource controls—such as quotas, dedicated node pools, or taints and tolerations—are needed to prevent contention and pod eviction loops.
Evaluating Rook-Ceph: Complete Capability Versus Operational Overhead
Rook functions as a cloud-native storage orchestrator designed to run Ceph clusters natively within Kubernetes. Per official Rook documentation, the orchestrator exposes three primary storage types: block (ReadWriteOnce), shared filesystem (ReadWriteMany via CephFS), and object storage through an S3-compatible endpoint. This makes it an exhaustive answer for teams needing unified storage from a single control plane. Teams can understand the differences between block, file, and object storage to determine whether such breadth is required.
However, Ceph's architecture demands significant operational discipline. Ceph integration documentation clarifies that Rook is strictly an orchestration tool that integrates with Ceph's orchestrator framework; Rook is not the storage engine itself. When underlying Placement Groups degrade or an OSD encounters low-level metadata corruption, your engineers are managing Ceph internals, not Kubernetes primitives.
Production readiness enforces strict infrastructure baselines. Rook documentation specifies that configuring a Ceph storage cluster requires at least one local storage option per node. Furthermore, Rook's quickstart guidance establishes that a healthy cluster requires all monitors to maintain quorum, an active manager, and at least three OSDs up and in. In terms of platform compatibility, current Rook documentation lists Kubernetes v1.31 through v1.37 as supported, which still requires continuous testing across Kubernetes minor version upgrades and does not guarantee identical behaviour for every Ceph feature in each release.
Evaluating Longhorn: Lightweight Simplicity and the V2 Engine
Longhorn approaches on-premise Kubernetes storage from the opposite design philosophy, positioning itself as a lightweight, reliable, and powerful distributed block storage system. Instead of deploying an expansive storage cluster that provides unified storage APIs, Longhorn focuses on block devices. It handles high availability by replicating each block volume synchronously across multiple replicas stored on separate nodes, establishing automated node failure handling directly inside its core design.
A critical milestone for the platform occurred on 2 June 2026, when the Longhorn v1.12.0 release notes marked the general availability of its V2 Data Engine. This evolution refines the underlying data engine and storage pipeline, improving performance characteristics and making Longhorn more attractive for higher‑throughput container workloads compared with earlier releases.
Day-2 operational burdens with Longhorn are generally lower than with Ceph because Longhorn is managed through native Kubernetes Custom Resource Definitions (CRDs) and an intuitive built-in graphical dashboard. However, because volume replication occurs synchronously over the standard Kubernetes node network, inter-node latency heavily governs write performance. Network congestion directly manifests as write degradation, demanding disciplined interface separation.
Enterprise Array CSI Drivers: Leveraging Existing Storage Fabrics
Organisations maintaining existing SAN or NAS arrays often find software-defined storage overlays inside Kubernetes redundant. Mature enterprise storage platforms provide dedicated CSI drivers tailored to their architectures: NetApp Trident connects Kubernetes directly to ONTAP systems for unified file and block via NFS or iSCSI; Dell Container Storage Modules (CSM) expose PowerStore, PowerMax, and PowerFlex arrays; Pure Storage delivers direct block integration via its FlashArray CSI; and the HPE CSI Driver for Kubernetes bridges Alletra and Primera platforms over Fibre Channel or iSCSI. Infrastructure teams can compare SAN, NAS, and DAS architectures for your environment before deciding whether to expose external volumes directly to container nodes.
In the UK market, procurement dynamics heavily favour this model when storage hardware is already capitalised or procured under structured frameworks. Public-sector buyers and NHS trusts routinely source enterprise arrays and vendor-backed support through Crown Commercial Service agreements such as the G-Cloud framework or Technology Products & Associated Services (TePAS). Leveraging these existing contracts and manufacturer SLAs shifts hardware lifecycle and break-fix liabilities away from internal platform engineers and onto established vendor support teams.
The operational challenge shifts from managing storage daemons to managing the CSI driver interface. Array CSI drivers require rigid synchronization between Kubernetes worker node host bus adapters (HBAs) or software iSCSI initiators and the array's target ports. While day-2 storage maintenance (such as non-disruptive array controller upgrades) is offloaded to enterprise storage teams, platform engineers must still track driver compatibility across Kubernetes releases.
Day-2 Operations, Failure Modes, and Resiliency Architecture
Selecting persistent storage based solely on installation simplicity is a common trap for infrastructure teams. Day-2 operations—handling node evictions, capacity rebalancing, cluster-wide upgrades, and unexpected power outages—dictate true total cost of ownership. When evaluating these options, failure modes must be prioritised over idealised benchmarks.
Longhorn confines failure domains to individual volume replica sets. If a node fails, Longhorn flags the impacted replica as degraded and immediately provisions a rebuild onto another node with available capacity. Rook-Ceph isolates failure using Ceph CRUSH maps; however, an unexpected loss of Ceph monitor quorum will halt I/O operations across the entire Kubernetes cluster, risking service outages across every attached pod.
Disaster recovery and backup workflows must conform to rigorous operational standards. The UK government’s National Cyber Security Centre (NCSC) guidance advises organisations to establish tested backup strategies and resilient defences against ransomware. Persistent volumes must support native point-in-time CSI volume snapshots that integrate seamlessly into immutable, offsite backup repositories to minimise both Recovery Point Objectives (RPO) and Recovery Time Objectives (RTO).
- •Ceph Monitor Quorum: Losing a majority of monitors freezes the entire cluster's storage plane until manual intervention restores consensus.
- •Longhorn Network Saturation: Volume rebuilding can consume massive host bandwidth, causing packet drops for co-located tenant pods unless rate limits are applied.
- •Array Multi-Attach Errors: Stale volume attachments during worker node panics can block pods from rescheduling until the array unmaps the target LUN.
View the data behind this chart
| Layer | Detail |
|---|---|
| Active Manager & CSI Plugins | Manages cluster state and mounts volumes to pods |
| Monitor Quorum (Ceph MONs) | Maintains distributed consensus and cluster map |
| Minimum 3 Ceph OSDs (Up and In) | Underlying storage daemons running on local disks |
UK Compliance, Security Posture, and Operational Governance
Deploying stateful infrastructure within the UK brings specific regulatory responsibilities. For public-sector contracts and enterprise supply chains, the UK government's Cyber Essentials scheme establishes an enforced baseline of five technical controls: firewalls, secure configuration, user access control, malware protection, and security update management. On-premise storage deployments must inherit these controls at every layer.
Storage platforms that run storage daemons directly on worker nodes—such as Rook/Ceph and Longhorn—tend to increase the attack surface because these processes usually need elevated privileges and direct access to physical block devices. Security teams must enforce strict role-based access control (RBAC) and segregate storage traffic onto dedicated VLANs, preventing multi-tenant workload pods from sniffing unencrypted replication traffic across the cluster network.
Array-backed CSI setups can help streamline aspects of compliance oversight when enterprise SANs are deployed on dedicated storage networks, because data access can be isolated from general-purpose application traffic. In contrast, distributed Kubernetes SDS engines require deliberate engineering to ensure at-rest encryption keys (managed via HashiCorp Vault or external KMS) and transit encryption (such as mTLS or WireGuard) comply with UK data protection and governance standards.
A Practical Implementation Guide: Rook-Ceph Cluster Verification
To illustrate how an open-source storage orchestrator operates on bare metal, consider deploying a production-grade Rook-Ceph cluster across three nodes. Because Rook documentation mandates at least one local storage option per node, raw, unformatted disks (such as NVMe drives or raw SATA SSDs) must be attached and discoverable across worker nodes before initialising the Ceph cluster CRD.
The deployment begins by installing the Rook operator CRDs and manifests into the rook-ceph namespace. Once the operator pod is running, the CephCluster custom resource is submitted. The operator discovers available disks across participating nodes and provisions the required Ceph daemons: Monitors (mon), Managers (mgr), and Object Storage Daemons (OSDs).
Verification is the final mandatory step before binding production workloads. Per Rook's quickstart documentation, a production cluster is verified as healthy only when all monitors achieve quorum, an active manager is running, and at least three OSDs are reported 'up' and 'in'. Once this state is validated via the Ceph CLI tools, a CephBlockPool and a corresponding StorageClass can be provisioned to serve PersistentVolumeClaims.
Decision Framework: Scenario-Based Storage Recommendations
Choosing the correct persistent storage architecture depends entirely on your existing infrastructure, team skillset, and operational requirements. For UK infrastructure buyers, the decisive issue is usually not the software licence but the people and outage cost: self-managed storage on Kubernetes often means paying for scarce Ceph/Rook or CSI expertise, plus the hardware and support burden when the platform team moves on. Assessing total cost of ownership requires balancing operational resilience against day-2 overhead: Ceph/Rook is the deepest option but the heaviest to run; Longhorn is simpler but demands disciplined replica management; and array CSI drivers reduce cluster-side complexity if an existing SAN/NAS is already well-supported and properly sized in the rack.
Best if you already run a modern enterprise SAN/NAS: Enterprise Array CSI Drivers. For organisations with existing NetApp, Dell, Pure, or HPE arrays and dedicated storage administrators, this path is the most operationally predictable. It eliminates container-level replication overhead, leverages existing hardware investments and SLA guarantees, and delegates hardware failure responses to established enterprise support contracts.
Best if you have no legacy array and need lightweight block storage: Longhorn (v1.12.0+). For teams lacking external arrays that require pure Kubernetes-native operations without deep distributed-systems overhead, Longhorn is the pragmatic choice. Its V2 Data Engine improves performance characteristics, its CRD-based architecture aligns with standard Kubernetes operational patterns, and its replica failure domains are straightforward to troubleshoot without specialist storage certifications.
Best if you need unified block, file, and object at scale: Rook-Ceph. For large-scale bare-metal estates requiring a unified platform across block (RBD), shared filesystem (CephFS), and S3-compatible object storage, Rook-Ceph is unmatched. However, it requires dedicated Ceph operational expertise and disciplined governance to safely manage cluster lifecycle events, Placement Group health, and software upgrades.
Sources
Every figure in this article traces to the sources below.
- •CNCF — 2025 Annual Cloud Native Survey Announcement
- •Rook — Quickstart Documentation & Cluster Health Requirements
- •Ceph — Ceph Orchestrator Rook Module Documentation
- •Longhorn — Architecture, Synchronous Replication & v1.12.0 Notes
- •Ofgem — Energy Price Cap July to September 2026 Announcement
- •Ofcom — Connected Nations Gigabit Broadband Availability
- •UK Government — Cyber Essentials Scheme Overview
- •NCSC — Guidance on Backup Strategies and Ransomware Defence