cluster.drgutman.org

← All notes

September 20, 2026 · David Gutman

Ceph Benchmarks Deep Dive

Real-world rados bench results for a heterogeneous Ceph cluster running HDD, SSD, NVMe, and Optane-backed OSDs — with 3× replication, MTU 9000, and the benchmark bugs that nearly sent us down the wrong path.

The Cluster

Our main Ceph cluster (pmoxprime) runs Ceph Squid 19.2.4 across 14 Proxmox nodes. It's the opposite of a uniform deployment — 99 OSDs spread across four device classes with a CRUSH hierarchy designed to keep each tier isolated:

TierOSDsDevicePoolsTarget Workload
HDD536–12 TB SATA (Seagate)ceph_bulk, cephfs_images_bulk, profile_hddCapacity — bulk storage, WSI archive
SSD283.49 TB SATA (Samsung/Micron)ceph_ssdBalanced — general-purpose VM storage
NVMe101.6 TB Intel P3600ceph_nvme, profile_nvmePerformance — low latency, high IOPS
Optane-Backed83.49 TB SATA SSD + 64 GB NVMe DBceph_optaneLow-latency — metadata-heavy workloads

The key architectural decision: every HDD has its WAL on NVMe via BlueFS dedicated DB. The SSDoptane class pairs a SATA SSD data device with a dedicated 64 GB NVMe partition for BlueFS DB, with WAL co-located. This offload architecture is what makes the HDD tier competitive for small writes.


Benchmark Methodology

All numbers here come from rados bench — Ceph's built-in benchmarking tool. Parameters:

⚠️ Important caveats: rados bench uses a single OSD thread per rados instance and limited queue depth. These numbers underreport what the drives can do compared to a properly multi-threaded workload. They're useful for comparing tiers, not for absolute drive benchmarking. True peak performance would be 2-4× higher with optimal queue depth and parallel rados instances.

Large Object Performance (4 MB)

For large sequential objects, the network is often the bottleneck, not the drives. Here's how each tier compares:

ClassWrite BWWrite IOPSRead BWRead IOPS
NVMe 1,175 MB/s 293 1,175 MB/s 294
SSD 431 MB/s 107 1,170 MB/s 292
HDD 346 MB/s 86 852 MB/s 213
📊 Key observation: SSD and NVMe reads are virtually identical at ~1,170 MB/s — both saturate the 25 GbE network. HDD reads at 852 MB/s, which is excellent for spinning rust and reflects the DB/WAL offload doing its job. Write throughput tells a different story: the NVMe tier writes at 2.7× the SSD tier, and the HDD tier holds its own at 346 MB/s — about 80% of the SSD write rate.

Small Object IOPS (4 KB) — Where Tiers Actually Separate

This is the test that reveals real performance differences. Small-object IOPS are dominated by seek time, controller overhead, and the effectiveness of the BlueFS DB/WAL offload:

ClassWrite BWWrite IOPSRead BWRead IOPS
NVMe 32.5 MB/s 8,320 125 MB/s 32,024
SSD 7.8 MB/s 1,999 147 MB/s 37,629
HDD 9.0 MB/s 2,299 94 MB/s 23,988
🔍 The HDD surprise: HDDs achieve 2,299 write IOPS — higher than SATA SSDs at 1,999 IOPS. This is entirely due to the NVMe WAL offload. The write penalty of spinning rust is almost eliminated because small writes land on the NVMe WAL and are committed asynchronously. For read IOPS, SSDs lead at 37,629 vs HDD's 23,988 — a narrower gap than you'd expect, again thanks to the NVMe DB accelerating metadata lookups.

Understanding the NVMe Write IOPS Number

8,320 write IOPS might seem low for NVMe. Here's the context:


Read Performance: Sequential vs Random

An interesting detail: Ceph's rados bench with sequential and random read flags showed identical performance for 4 MB objects across all tiers. This makes sense — at 4 MB objects with 3× replication, Ceph's CRUSH distribution spreads the replicas across OSDs such that sequential access patterns don't give the read head a meaningful advantage. The bottleneck is network throughput, not disk seek:

ClassSeq Read (4 MB)Rand Read (4 MB)Difference
NVMe1,175 MB/s1,175 MB/s0%
SSD1,170 MB/s1,170 MB/s0%
HDD852 MB/s852 MB/s0%

For 4 KB objects, random reads were slightly slower than sequential — the HDD tier dropped from ~24K to ~19K IOPS — but the gap was smaller than we expected, again reflecting the DB/WAL offload absorbing the rotational penalty.


The Benchmark Bug That Almost Derailed Us

An important meta-lesson: our automated benchmark script was wrong for weeks. The script parsed rados bench output looking for "IOPS:", but the actual output uses "Average IOPS:". Every IOPS value the script reported was silently zero or incorrect.

We discovered this when manual rados bench runs gave completely different — and much better — numbers. The lesson: always verify automated benchmark results with manual runs. A parsing bug can hide for months if nobody questions the output.

The corrected numbers told a much better story. The HDD tier went from looking mediocre to genuinely impressive once the DB/WAL offload was properly reflected in the IOPS figures.


MTU 9000: The 10× Improvement

Early in the cluster's life, we ran with standard MTU 1500 on the Ceph network. After switching to MTU 9000 (jumbo frames), we saw dramatic improvements — roughly 10× for small-block workloads and significant gains for large-block transfers.

The mechanism: with MTU 9000, each Ethernet frame carries 6× more payload. For a 4 MB write, that's ~290 frames instead of ~2,800 at MTU 1500. Less CPU time spent on interrupt handling, fewer context switches, and the 25 GbE links can actually reach line rate instead of being frame-bound.

Every Ceph node in our cluster now has MTU 9000 on dedicated storage dedicated storage VLANs (separate public and cluster networks). The one lesson: verify end-to-end — a single misconfigured switch port will cause packet loss and performance degradation that's hard to diagnose.


Optane-Backed Tier: Work in Progress

The SSDoptane class (8 OSDs) pairs SATA SSDs with dedicated 64 GB NVMe partitions for BlueFS DB. We haven't published benchmark numbers for this tier yet because early tests revealed configuration issues with the pool that skewed results. The pool is being rebuilt for proper characterization.

What we expect: the 64 GB NVMe DB device should accelerate metadata operations and small writes dramatically compared to the plain SSD tier, while the SATA SSD provides 3.49 TB of capacity per OSD. This tier should slot between SSD and NVMe in the latency hierarchy — close to NVMe for metadata-heavy workloads, close to SSD for bulk throughput.


Cluster-Wide Summary

MetricValue
Ceph VersionSquid 19.2.4
Total OSDs99 (53 HDD + 28 SSD + 10 NVMe + 8 Optane-backed)
Raw Capacity~451 TiB total, ~345 TiB available
Placement Groups1,601 (all active+clean)
Pools12 (5 replicated + 1 erasure-coded + profile/test pools)
CRUSH Rules9 (4 device-class + 3 replicated + 2 EC)
Network25 GbE with MTU 9000
Best 4 MB Write1,175 MB/s (NVMe)
Best 4 KB Write IOPS8,320 (NVMe)
Best 4 KB Read IOPS37,629 (SSD)

What We'd Do Differently

  1. Higher PG counts. 32 PGs for 8 NVMe OSDs leaves performance on the table. 128–256 PGs would improve parallelism, especially for small-object workloads.
  2. Benchmark with proper queue depth. rados bench at 16 threads underreports NVMe capability. A multi-instance fio workload via rbd map or librbd would give more realistic numbers.
  3. Standardize MTU 9000 everywhere. Not all nodes have the Ceph public VLAN configured — some fall back to the management network at MTU 1500, creating a mixed-MTU situation.
  4. Fix the benchmark script early. Weeks of development decisions were based on bad data because a grep string was wrong.
  5. Characterize the Optane tier before putting it in production. We rushed the deployment and discovered config issues during benchmarking that should have been caught earlier.

How to Run This Yourself

# Quick rados bench on any Ceph pool
rados -p <pool> bench 30 write -b 4194304 --no-cleanup
rados -p <pool> bench 30 seq -b 4194304
rados -p <pool> bench 30 rand -b 4194304

# 4K small-object test
rados -p <pool> bench 30 write -b 4096 --no-cleanup
rados -p <pool> bench 30 rand -b 4096

# Clean up test objects
rados -p <pool> cleanup

For a comprehensive campaign across multiple tiers, our INFRA-BRAIN repository has a benchmark toolkit with automated campaign scripts.


Related Reading


Originally published in the GutmanLab internal documentation. Benchmarks measured September 2026; see the benchmark index for current caveats.