Field notes
Longer write-ups that do not fit in a table. These are the stories behind the benchmark numbers: what broke, what we changed, and what we would do differently.
Ceph benchmarks deep dive
Real-world
Read the deep dive →
rados bench results for a heterogeneous Ceph cluster — HDD, SSD, NVMe, and
Optane-backed OSDs — with 3× replication and MTU 9000. Includes the parsing bug that reported
wrong IOPS for weeks, the MTU 9000 improvement, and why the HDD tier is more competitive than it
has any right to be.
How 3 ZFS settings cost us 11× performance
A 503 GiB TrueNAS host with 8 SSDs in mirror, an NVMe special vdev, and a SLOG was writing at
375 MiB/s.
Read the tuning story →
special_small_blocks, logbias, and zfs_dirty_data_max
were each compounding the problem. After three changes: 4,271 MiB/s.
More to come
The Optane-backed Ceph tier is being rebuilt and characterized; its benchmark note will land here when the numbers are trustworthy. The cluster status and hardware pages are updated as the system changes.