Benchmarks
The most useful thing about a heterogeneous cluster is that you can measure what each kind of hardware actually does under the same workload. These are production numbers, not clean-room numbers: 3× replication, live cluster, September 2026.
Methodology
- Ceph:
rados bench, 30 seconds per test, 16 concurrent threads, 3× replication (size 3, min_size 2), MTU 9000 on the storage fabric, run from a monitor node against dedicated profile pools. - TrueNAS:
fioruns against each pool, sequential and 4 KB random, plus a 70/30 mixed workload. ARC hit rates recorded per pool. - Both: results reflect a real cluster doing real work. Background scrub, backup traffic, and other tenants can move the numbers by 10–20%.
Read this before quoting a number
rados bench uses a limited queue depth and a single OSD thread per RADOS instance. It underreports NVMe and multi-drive capability, often by 2–4×. Use these figures to compare tiers against each other, not as drive datasheet replacements. The NVMe write number in particular is a benchmark-harness ceiling, not a drive ceiling.
1. Ceph — large objects (4 MB)
At 4 MB, the network is usually the limit, not the media. Both SSD and NVMe reads land at roughly 1,170 MB/s, which is what a 25 GbE link can carry after overhead. The HDD tier's 852 MB/s read is the interesting result: spinning rust with NVMe WAL/DB offload is not the bottleneck people expect.
| Class | Write BW | Write IOPS | Read BW | Read IOPS | Rand read BW | Rand read IOPS |
|---|---|---|---|---|---|---|
| NVMe | 1,175 MB/s | 293 | 1,175 MB/s | 294 | 1,175 MB/s | 294 |
| SSD | 431 MB/s | 107 | 1,170 MB/s | 292 | 1,170 MB/s | 292 |
| HDD + NVMe WAL | 346 MB/s | 86 | 852 MB/s | 213 | 852 MB/s | 213 |
4 MB sequential read
Random and sequential reads were effectively identical at 4 MB. With three replicas placed across OSDs by CRUSH, a 4 MB object is large enough that seek behavior stops mattering; throughput is set by the network and the slowest replica.
Write throughput separates the tiers more honestly: NVMe writes at 2.7× the SATA SSD rate, and the HDD tier reaches about 80% of the SSD rate. That last figure is entirely due to BlueFS WAL offload.
2. Ceph — small objects (4 KB)
This is the test that reveals real differences. Small-object IOPS are dominated by seek time, controller overhead, and whether metadata operations land on flash.
| Class | Write BW | Write IOPS | Read BW | Read IOPS |
|---|---|---|---|---|
| NVMe | 32.5 MB/s | 8,320 | 125 MB/s | 32,024 |
| SSD | 7.8 MB/s | 1,999 | 147 MB/s | 37,629 |
| HDD + NVMe WAL | 9.0 MB/s | 2,299 | 94 MB/s | 23,988 |
HDD write IOPS beat SATA SSD write IOPS
The HDD tier reached 2,299 4 KB write IOPS against 1,999 for the plain SATA SSD tier. The NVMe WAL absorbs the small writes and commits them asynchronously, so the rotating media no longer pays the seek penalty for each one. Reads still favor SSD (37,629 vs 23,988 IOPS), but the gap is far smaller than the hardware suggests.
Why the NVMe write number looks low
8,320 write IOPS is unimpressive for NVMe, and it is a harness artifact rather than a drive limit:
- 3× replication means each logical write becomes three OSD writes; the raw per-drive figure is roughly 25,000 IOPS.
- 16 rados threads is far below the queue depth these drives can sustain (64–128+).
- 32 PGs across 8 OSDs limits parallelism; 128–256 PGs would distribute load better.
- The drives are Intel P3600 — PCIe 3.0 enterprise NVMe from 2015, rated around 2,600 MB/s sequential and 460K random-read IOPS at QD128. Fast for their generation, not modern Gen 4.
3. Optane-backed tier — pending
No published numbers yet
The ceph_optane tier pairs a 3.49 TB SATA SSD with a dedicated 64 GB NVMe partition for BlueFS DB. Early benchmark runs exposed configuration problems that skewed the results, so the pool is being rebuilt before it is characterized. The expectation is a tier between SSD and NVMe: close to NVMe for metadata-heavy work, close to SSD for bulk throughput. We would rather publish nothing than publish a number we do not trust.
4. TrueNAS — four pools, four personalities
The lab also runs four TrueNAS SCALE 25.04 hosts. Each pool has a different architecture, and the benchmark results track those choices closely.
| Pool / host | Architecture | ARC hit | Seq write | Seq read | 4K rand read | Mixed 70/30 |
|---|---|---|---|---|---|---|
| daghome icarus, post-tuning |
4× mirror + NVMe special + SLOG | 94.1% | 4,271 MiB/s | 4,430 MiB/s | 621K IOPS | 54,800 R / 23,500 W |
| samsungssd scratchy |
3× mirror + NVMe special | 99.7% | 1,413 MiB/s | 6,403 MiB/s | 253K IOPS | 29,600 R / 12,700 W |
| slidevault brainhost |
2× 6-wide RAIDZ2 + NVMe special + SLOG | 96.1% | 1,405 MiB/s | 6,675 MiB/s | 237K IOPS | 26,600 R / 11,400 W |
| tank gauss |
2× 6-wide RAIDZ2 + NVMe special | 98.9% | 905 MiB/s | 6,420 MiB/s | 101K IOPS | 15,100 R / 6,477 W |
Sequential read
slidevault pool reads at 6.7 GB/s thanks to two RAIDZ2 groups and NVMe special vdevs.Random read IOPS
daghome, an all-SSD pool with an NVMe special vdev, is the winner. A 94% ARC hit rate plus flash makes it excellent for metadata-heavy and home-directory workloads.Mixed workload
daghome sustains 54.8K read + 23.5K write IOPS at a 70/30 mix after tuning — enough for interactive home directories and compute scratch.Small-file write
scratchy's all-SSD mirror pool plus NVMe special reached 396,000 small-file write IOPS at 1,546 MiB/s — the right tool for temp and scratch traffic.Takeaway
NVMe special vdevs and high ARC hit rates make even HDD pools feel all-flash for reads. Write performance is usually a configuration problem before it is a hardware problem — the Icarus story is the proof.
5. The Icarus tuning story: 375 → 4,271 MiB/s
The daghome pool had 8 SATA SSDs in four mirrors, a 2× NVMe special vdev, a 120 GB Intel NVMe SLOG, and 503 GiB of RAM. Sequential writes should have been measured in gigabytes per second. They measured 375 MiB/s.
Bottleneck 1: special_small_blocks=128K with recordsize=16K
The special vdev is meant for metadata and small-file data. The threshold was set to 128 KB, but the dataset's recordsize was 16 KB. Every 16 KB block was therefore diverted to the 2× NVMe special mirror. The 8 SATA SSDs sat idle for writes while the metadata accelerator became the main write path.
Fix: special_small_blocks=0. Metadata stays on NVMe; data spreads across the main pool. Result: 375 → 478 MiB/s (+27%).
Bottleneck 2: logbias=latency
This forces every write — including asynchronous writes that should aggregate in RAM — through the ZIL/SLOG. The SLOG was a single 120 GB NVMe with roughly 360 MB/s of sequential write throughput, so it imposed a hard ceiling below what the main pool could do.
Fix: logbias=throughput. Async writes bypass the ZIL. Result: 375 → 524 MiB/s (+40% cumulative).
Bottleneck 3: zfs_dirty_data_max=4 GB
OpenZFS traditionally defaults this to about 10% of RAM — roughly 50 GB here. TrueNAS SCALE ships a hard-coded 4 GB, inherited from appliance-era defaults. With 503 GiB of RAM and a 4 GB cap, the transaction group throttle engaged every few seconds and blocked writes that the SSDs could easily absorb.
Fix: raise it to 32 GB (about 6% of RAM; 10–20% is reasonable for write-heavy pools). Result: 524 → 4,271 MiB/s (+1,039% cumulative). The NFS-like psync workload reached 1,549 MiB/s, enough to saturate 25 GbE.
Eleven times faster with no hardware changes
Three settings. No new drives, no new controller, no rebuild. The pool was not slow; it was configured to be slow. This is the single most reusable lesson on the site.
Read the full Icarus write-up, including the fio commands and ZFS reasoning →
6. Lessons and caveats
MTU 9000 is not optional
BlueFS offload is a force multiplier
Benchmark scripts lie
"IOPS:" while rados bench prints "Average IOPS:". Every reported IOPS value was silently wrong for weeks. Manual runs exposed it and told a much better story.Test the real access pattern
direct=1 bypasses the page cache; psync approximates NFS. The two can differ by 3×. Benchmark the workload you actually serve, not the one that produces the prettiest chart.What we would do differently
- Raise PG counts on the NVMe pools (128–256 instead of 32) to improve small-object parallelism.
- Benchmark NVMe with realistic queue depth via
rbd mapor multi-instancefio, not just 16-threadrados bench. - Standardize MTU 9000 on every Ceph path; mixed-MTU links are hard to diagnose and quietly expensive.
- Validate the benchmark harness before trusting months of results.
- Characterize a new tier before putting production data on it (see the Optane tier).
Reproduce it
# 4 MB large-object test
rados -p <pool> bench 30 write -b 4194304 --no-cleanup
rados -p <pool> bench 30 seq -b 4194304
rados -p <pool> bench 30 rand -b 4194304
# 4 KB small-object test
rados -p <pool> bench 30 write -b 4096 --no-cleanup
rados -p <pool> bench 30 rand -b 4096
rados -p <pool> cleanup
Always clean up the benchmark objects when you are done.
Download the numbers
The tables on this page are mirrored as CSV so they can be reused or re-plotted. They preserve the September 2026 values exactly, including the caveats above.
- Ceph rados bench — 4 MB and 4 KB by tier
- TrueNAS fio — four pools, sequential/random/mixed
- Icarus ZFS tuning — progressive results
If you cite these numbers, cite them as a September 2026 snapshot of a shared production cluster, not as isolated drive benchmarks.