Ceph Benchmarks Deep Dive
Real-world rados bench results for a heterogeneous Ceph cluster running HDD, SSD, NVMe, and Optane-backed OSDs — with 3× replication, MTU 9000, and the benchmark bugs that nearly sent us down the wrong path.
The Cluster
Our main Ceph cluster (pmoxprime) runs Ceph Squid 19.2.4 across 14 Proxmox nodes. It's the opposite of a uniform deployment — 99 OSDs spread across four device classes with a CRUSH hierarchy designed to keep each tier isolated:
| Tier | OSDs | Device | Pools | Target Workload |
|---|---|---|---|---|
| HDD | 53 | 6–12 TB SATA (Seagate) | ceph_bulk, cephfs_images_bulk, profile_hdd | Capacity — bulk storage, WSI archive |
| SSD | 28 | 3.49 TB SATA (Samsung/Micron) | ceph_ssd | Balanced — general-purpose VM storage |
| NVMe | 10 | 1.6 TB Intel P3600 | ceph_nvme, profile_nvme | Performance — low latency, high IOPS |
| Optane-Backed | 8 | 3.49 TB SATA SSD + 64 GB NVMe DB | ceph_optane | Low-latency — metadata-heavy workloads |
The key architectural decision: every HDD has its WAL on NVMe via BlueFS dedicated DB. The SSDoptane class pairs a SATA SSD data device with a dedicated 64 GB NVMe partition for BlueFS DB, with WAL co-located. This offload architecture is what makes the HDD tier competitive for small writes.
Benchmark Methodology
All numbers here come from rados bench — Ceph's built-in
benchmarking tool. Parameters:
- 30-second test duration per object size
- 16 concurrent threads (limited by rados bench defaults)
- 3× replication across all pools (size=3, min_size=2)
- MTU 9000 on the Ceph public and cluster networks
- Tests run from a monitor node (triton) against dedicated profile pools
Large Object Performance (4 MB)
For large sequential objects, the network is often the bottleneck, not the drives. Here's how each tier compares:
| Class | Write BW | Write IOPS | Read BW | Read IOPS |
|---|---|---|---|---|
| NVMe | 1,175 MB/s | 293 | 1,175 MB/s | 294 |
| SSD | 431 MB/s | 107 | 1,170 MB/s | 292 |
| HDD | 346 MB/s | 86 | 852 MB/s | 213 |
Small Object IOPS (4 KB) — Where Tiers Actually Separate
This is the test that reveals real performance differences. Small-object IOPS are dominated by seek time, controller overhead, and the effectiveness of the BlueFS DB/WAL offload:
| Class | Write BW | Write IOPS | Read BW | Read IOPS |
|---|---|---|---|---|
| NVMe | 32.5 MB/s | 8,320 | 125 MB/s | 32,024 |
| SSD | 7.8 MB/s | 1,999 | 147 MB/s | 37,629 |
| HDD | 9.0 MB/s | 2,299 | 94 MB/s | 23,988 |
Understanding the NVMe Write IOPS Number
8,320 write IOPS might seem low for NVMe. Here's the context:
- 3× replication: every write to Ceph is stored on three OSDs. The raw drive write IOPS is ~25,000 per drive — reasonable for Intel P3600 (PCIe 3.0) drives from 2015.
- rados bench at 16 threads is the bottleneck here. These NVMe drives can handle queue depths of 64–128+. With a properly tuned workload, expect 2-3× these numbers.
- 32 PGs for 8 OSDs limits parallelism. Increasing to 128–256 PGs would distribute load better.
- The Intel P3600 is a PCIe 3.0 enterprise NVMe drive — good for its era (~2,600 MB/s sequential, ~460K random read IOPS at QD128), but not a modern Gen 4 drive.
Read Performance: Sequential vs Random
An interesting detail: Ceph's rados bench with sequential and
random read flags showed identical performance for 4 MB
objects across all tiers. This makes sense — at 4 MB objects with 3×
replication, Ceph's CRUSH distribution spreads the replicas across OSDs
such that sequential access patterns don't give the read head a meaningful
advantage. The bottleneck is network throughput, not disk seek:
| Class | Seq Read (4 MB) | Rand Read (4 MB) | Difference |
|---|---|---|---|
| NVMe | 1,175 MB/s | 1,175 MB/s | 0% |
| SSD | 1,170 MB/s | 1,170 MB/s | 0% |
| HDD | 852 MB/s | 852 MB/s | 0% |
For 4 KB objects, random reads were slightly slower than sequential — the HDD tier dropped from ~24K to ~19K IOPS — but the gap was smaller than we expected, again reflecting the DB/WAL offload absorbing the rotational penalty.
The Benchmark Bug That Almost Derailed Us
An important meta-lesson: our automated benchmark script was wrong
for weeks. The script parsed rados bench output looking for
"IOPS:", but the actual output uses "Average IOPS:".
Every IOPS value the script reported was silently zero or incorrect.
We discovered this when manual rados bench runs gave completely
different — and much better — numbers. The lesson: always verify
automated benchmark results with manual runs. A parsing bug can
hide for months if nobody questions the output.
The corrected numbers told a much better story. The HDD tier went from looking mediocre to genuinely impressive once the DB/WAL offload was properly reflected in the IOPS figures.
MTU 9000: The 10× Improvement
Early in the cluster's life, we ran with standard MTU 1500 on the Ceph network. After switching to MTU 9000 (jumbo frames), we saw dramatic improvements — roughly 10× for small-block workloads and significant gains for large-block transfers.
The mechanism: with MTU 9000, each Ethernet frame carries 6× more payload. For a 4 MB write, that's ~290 frames instead of ~2,800 at MTU 1500. Less CPU time spent on interrupt handling, fewer context switches, and the 25 GbE links can actually reach line rate instead of being frame-bound.
Every Ceph node in our cluster now has MTU 9000 on dedicated storage dedicated storage VLANs (separate public and cluster networks). The one lesson: verify end-to-end — a single misconfigured switch port will cause packet loss and performance degradation that's hard to diagnose.
Optane-Backed Tier: Work in Progress
The SSDoptane class (8 OSDs) pairs SATA SSDs with dedicated 64 GB NVMe partitions for BlueFS DB. We haven't published benchmark numbers for this tier yet because early tests revealed configuration issues with the pool that skewed results. The pool is being rebuilt for proper characterization.
What we expect: the 64 GB NVMe DB device should accelerate metadata operations and small writes dramatically compared to the plain SSD tier, while the SATA SSD provides 3.49 TB of capacity per OSD. This tier should slot between SSD and NVMe in the latency hierarchy — close to NVMe for metadata-heavy workloads, close to SSD for bulk throughput.
Cluster-Wide Summary
| Metric | Value |
|---|---|
| Ceph Version | Squid 19.2.4 |
| Total OSDs | 99 (53 HDD + 28 SSD + 10 NVMe + 8 Optane-backed) |
| Raw Capacity | ~451 TiB total, ~345 TiB available |
| Placement Groups | 1,601 (all active+clean) |
| Pools | 12 (5 replicated + 1 erasure-coded + profile/test pools) |
| CRUSH Rules | 9 (4 device-class + 3 replicated + 2 EC) |
| Network | 25 GbE with MTU 9000 |
| Best 4 MB Write | 1,175 MB/s (NVMe) |
| Best 4 KB Write IOPS | 8,320 (NVMe) |
| Best 4 KB Read IOPS | 37,629 (SSD) |
What We'd Do Differently
- Higher PG counts. 32 PGs for 8 NVMe OSDs leaves performance on the table. 128–256 PGs would improve parallelism, especially for small-object workloads.
-
Benchmark with proper queue depth. rados bench at
16 threads underreports NVMe capability. A multi-instance fio workload
via
rbd mapor librbd would give more realistic numbers. - Standardize MTU 9000 everywhere. Not all nodes have the Ceph public VLAN configured — some fall back to the management network at MTU 1500, creating a mixed-MTU situation.
- Fix the benchmark script early. Weeks of development decisions were based on bad data because a grep string was wrong.
- Characterize the Optane tier before putting it in production. We rushed the deployment and discovered config issues during benchmarking that should have been caught earlier.
How to Run This Yourself
# Quick rados bench on any Ceph pool
rados -p <pool> bench 30 write -b 4194304 --no-cleanup
rados -p <pool> bench 30 seq -b 4194304
rados -p <pool> bench 30 rand -b 4194304
# 4K small-object test
rados -p <pool> bench 30 write -b 4096 --no-cleanup
rados -p <pool> bench 30 rand -b 4096
# Clean up test objects
rados -p <pool> cleanup
For a comprehensive campaign across multiple tiers, our INFRA-BRAIN repository has a benchmark toolkit with automated campaign scripts.
Related Reading
- The Pathology Frankencluster — full hardware inventory, network topology, and all storage tiers
- How 3 ZFS Settings Cost Us 11× Performance — our TrueNAS tuning story with lessons for any ZFS deployment
Originally published in the GutmanLab internal documentation. Benchmarks measured September 2026; see the benchmark index for current caveats.