cluster.drgutman.org

Benchmarks

The most useful thing about a heterogeneous cluster is that you can measure what each kind of hardware actually does under the same workload. These are production numbers, not clean-room numbers: 3× replication, live cluster, September 2026.

Methodology

  • Ceph: rados bench, 30 seconds per test, 16 concurrent threads, 3× replication (size 3, min_size 2), MTU 9000 on the storage fabric, run from a monitor node against dedicated profile pools.
  • TrueNAS: fio runs against each pool, sequential and 4 KB random, plus a 70/30 mixed workload. ARC hit rates recorded per pool.
  • Both: results reflect a real cluster doing real work. Background scrub, backup traffic, and other tenants can move the numbers by 10–20%.

Read this before quoting a number

rados bench uses a limited queue depth and a single OSD thread per RADOS instance. It underreports NVMe and multi-drive capability, often by 2–4×. Use these figures to compare tiers against each other, not as drive datasheet replacements. The NVMe write number in particular is a benchmark-harness ceiling, not a drive ceiling.

1. Ceph — large objects (4 MB)

At 4 MB, the network is usually the limit, not the media. Both SSD and NVMe reads land at roughly 1,170 MB/s, which is what a 25 GbE link can carry after overhead. The HDD tier's 852 MB/s read is the interesting result: spinning rust with NVMe WAL/DB offload is not the bottleneck people expect.

ClassWrite BWWrite IOPSRead BWRead IOPSRand read BWRand read IOPS
NVMe1,175 MB/s2931,175 MB/s2941,175 MB/s294
SSD431 MB/s1071,170 MB/s2921,170 MB/s292
HDD + NVMe WAL346 MB/s86852 MB/s213852 MB/s213

4 MB sequential read

NVMe
1,175 MB/s
SSD
1,170 MB/s
HDD
852 MB/s

Random and sequential reads were effectively identical at 4 MB. With three replicas placed across OSDs by CRUSH, a 4 MB object is large enough that seek behavior stops mattering; throughput is set by the network and the slowest replica.

Write throughput separates the tiers more honestly: NVMe writes at 2.7× the SATA SSD rate, and the HDD tier reaches about 80% of the SSD rate. That last figure is entirely due to BlueFS WAL offload.

2. Ceph — small objects (4 KB)

This is the test that reveals real differences. Small-object IOPS are dominated by seek time, controller overhead, and whether metadata operations land on flash.

ClassWrite BWWrite IOPSRead BWRead IOPS
NVMe32.5 MB/s8,320125 MB/s32,024
SSD7.8 MB/s1,999147 MB/s37,629
HDD + NVMe WAL9.0 MB/s2,29994 MB/s23,988

HDD write IOPS beat SATA SSD write IOPS

The HDD tier reached 2,299 4 KB write IOPS against 1,999 for the plain SATA SSD tier. The NVMe WAL absorbs the small writes and commits them asynchronously, so the rotating media no longer pays the seek penalty for each one. Reads still favor SSD (37,629 vs 23,988 IOPS), but the gap is far smaller than the hardware suggests.

Why the NVMe write number looks low

8,320 write IOPS is unimpressive for NVMe, and it is a harness artifact rather than a drive limit:

3. Optane-backed tier — pending

No published numbers yet

The ceph_optane tier pairs a 3.49 TB SATA SSD with a dedicated 64 GB NVMe partition for BlueFS DB. Early benchmark runs exposed configuration problems that skewed the results, so the pool is being rebuilt before it is characterized. The expectation is a tier between SSD and NVMe: close to NVMe for metadata-heavy work, close to SSD for bulk throughput. We would rather publish nothing than publish a number we do not trust.

4. TrueNAS — four pools, four personalities

The lab also runs four TrueNAS SCALE 25.04 hosts. Each pool has a different architecture, and the benchmark results track those choices closely.

Pool / hostArchitectureARC hitSeq writeSeq read4K rand readMixed 70/30
daghome
icarus, post-tuning
4× mirror + NVMe special + SLOG 94.1% 4,271 MiB/s 4,430 MiB/s 621K IOPS 54,800 R / 23,500 W
samsungssd
scratchy
3× mirror + NVMe special 99.7% 1,413 MiB/s 6,403 MiB/s 253K IOPS 29,600 R / 12,700 W
slidevault
brainhost
2× 6-wide RAIDZ2 + NVMe special + SLOG 96.1% 1,405 MiB/s 6,675 MiB/s 237K IOPS 26,600 R / 11,400 W
tank
gauss
2× 6-wide RAIDZ2 + NVMe special 98.9% 905 MiB/s 6,420 MiB/s 101K IOPS 15,100 R / 6,477 W

Sequential read

up to 6.7 GB/s
Every pool saturates or approaches its network link for large reads. Even the HDD-based slidevault pool reads at 6.7 GB/s thanks to two RAIDZ2 groups and NVMe special vdevs.

Random read IOPS

621K IOPS
daghome, an all-SSD pool with an NVMe special vdev, is the winner. A 94% ARC hit rate plus flash makes it excellent for metadata-heavy and home-directory workloads.

Mixed workload

78K total IOPS
daghome sustains 54.8K read + 23.5K write IOPS at a 70/30 mix after tuning — enough for interactive home directories and compute scratch.

Small-file write

396K IOPS
scratchy's all-SSD mirror pool plus NVMe special reached 396,000 small-file write IOPS at 1,546 MiB/s — the right tool for temp and scratch traffic.

Takeaway

NVMe special vdevs and high ARC hit rates make even HDD pools feel all-flash for reads. Write performance is usually a configuration problem before it is a hardware problem — the Icarus story is the proof.

5. The Icarus tuning story: 375 → 4,271 MiB/s

The daghome pool had 8 SATA SSDs in four mirrors, a 2× NVMe special vdev, a 120 GB Intel NVMe SLOG, and 503 GiB of RAM. Sequential writes should have been measured in gigabytes per second. They measured 375 MiB/s.

375
Baseline (MiB/s)
478
+ special_small_blocks=0
524
+ logbias=throughput
568
+ dirty_data=8 GB
4,271
+ dirty_data=32 GB

Bottleneck 1: special_small_blocks=128K with recordsize=16K

The special vdev is meant for metadata and small-file data. The threshold was set to 128 KB, but the dataset's recordsize was 16 KB. Every 16 KB block was therefore diverted to the 2× NVMe special mirror. The 8 SATA SSDs sat idle for writes while the metadata accelerator became the main write path.

Fix: special_small_blocks=0. Metadata stays on NVMe; data spreads across the main pool. Result: 375 → 478 MiB/s (+27%).

Bottleneck 2: logbias=latency

This forces every write — including asynchronous writes that should aggregate in RAM — through the ZIL/SLOG. The SLOG was a single 120 GB NVMe with roughly 360 MB/s of sequential write throughput, so it imposed a hard ceiling below what the main pool could do.

Fix: logbias=throughput. Async writes bypass the ZIL. Result: 375 → 524 MiB/s (+40% cumulative).

Bottleneck 3: zfs_dirty_data_max=4 GB

OpenZFS traditionally defaults this to about 10% of RAM — roughly 50 GB here. TrueNAS SCALE ships a hard-coded 4 GB, inherited from appliance-era defaults. With 503 GiB of RAM and a 4 GB cap, the transaction group throttle engaged every few seconds and blocked writes that the SSDs could easily absorb.

Fix: raise it to 32 GB (about 6% of RAM; 10–20% is reasonable for write-heavy pools). Result: 524 → 4,271 MiB/s (+1,039% cumulative). The NFS-like psync workload reached 1,549 MiB/s, enough to saturate 25 GbE.

Eleven times faster with no hardware changes

Three settings. No new drives, no new controller, no rebuild. The pool was not slow; it was configured to be slow. This is the single most reusable lesson on the site.

Read the full Icarus write-up, including the fio commands and ZFS reasoning →

6. Lessons and caveats

MTU 9000 is not optional

Jumbo frames on the storage fabric produced roughly a 10× gain in some small-block tests. With 9,000-byte frames a 4 MB write is ~290 frames instead of ~2,800. One misconfigured switch port can undo it for the whole tier.

BlueFS offload is a force multiplier

Putting WAL (and sometimes DB) on NVMe makes HDDs competitive for small writes. It is the difference between "spinning rust is for cold data" and 2,299 write IOPS.

Benchmark scripts lie

The automated harness parsed "IOPS:" while rados bench prints "Average IOPS:". Every reported IOPS value was silently wrong for weeks. Manual runs exposed it and told a much better story.

Test the real access pattern

direct=1 bypasses the page cache; psync approximates NFS. The two can differ by 3×. Benchmark the workload you actually serve, not the one that produces the prettiest chart.

What we would do differently

  1. Raise PG counts on the NVMe pools (128–256 instead of 32) to improve small-object parallelism.
  2. Benchmark NVMe with realistic queue depth via rbd map or multi-instance fio, not just 16-thread rados bench.
  3. Standardize MTU 9000 on every Ceph path; mixed-MTU links are hard to diagnose and quietly expensive.
  4. Validate the benchmark harness before trusting months of results.
  5. Characterize a new tier before putting production data on it (see the Optane tier).

Reproduce it

# 4 MB large-object test
rados -p <pool> bench 30 write -b 4194304 --no-cleanup
rados -p <pool> bench 30 seq -b 4194304
rados -p <pool> bench 30 rand -b 4194304

# 4 KB small-object test
rados -p <pool> bench 30 write -b 4096 --no-cleanup
rados -p <pool> bench 30 rand -b 4096

rados -p <pool> cleanup

Always clean up the benchmark objects when you are done.

Download the numbers

The tables on this page are mirrored as CSV so they can be reused or re-plotted. They preserve the September 2026 values exactly, including the caveats above.

If you cite these numbers, cite them as a September 2026 snapshot of a shared production cluster, not as isolated drive benchmarks.