cluster.drgutman.org

The Pathology Frankencluster

Fifteen Proxmox nodes. Four storage classes in one Ceph cluster. A second all-flash cluster for disaster recovery. None of it uniform, and that is the point.

15
Proxmox nodes
104
Ceph OSDs
531 TiB
Raw Ceph capacity
4
Storage tiers
9
DR cluster nodes

What this site is

This is the public notebook for the compute and storage behind GutmanLab's pathology imaging work at Emory. It is part hardware tour, part operations log, and part benchmark archive. The cluster was not purchased as a single system; it accumulated. Each node has its own generation, RAM speed, drive mix, and personality.

Two clusters are documented here:

pmoxprime — the main cluster

15 nodes · Ceph Squid
Xeon Gold / mixed generations · 25 GbE + 10 GbE · ~5.3 TiB RAM
Four Ceph device classes — HDD, SATA SSD, NVMe, and Optane-backed SSD — isolated by CRUSH rules. Pools are mapped to workloads instead of pretending every disk is interchangeable. This is where the Digital Slide Archive data, VMs, and research compute live.
Hardware and Ceph layout →

kubertpmox — the DR cluster

9 nodes · all-flash Ceph
~560 vCPU · ~2.7 TB RAM · PVE 9.1.2
A separate Proxmox cluster used for disaster recovery and staging. Thirteen OSDs across NVMe and enterprise SSD, a primary ceph_nvme pool, CephFS with standby MDS, and backups to a Proxmox Backup Server on the lab network.
Inside kubertpmox →

Why "Frankencluster"

Most clusters are bought in one purchase order: identical nodes, identical drives, identical warranty dates. This one grew from hand-me-downs, surplus finds, grant leftovers, and temporary measures that became permanent. Three generations of server hardware share one Ceph cluster. RAM runs at 2133, 2400, and 2666 MT/s. Drive classes span 6 TB spinning disks to Intel Optane-accelerated SSDs.

The interesting engineering question is not "can heterogeneous hardware work?" It obviously can. The question is which workload should land on which tier, and how much performance a small configuration change can unlock. That is what the benchmark section is for.

The short version of what we learned

  • CRUSH device classes are the abstraction that makes this sane. One cluster, four tiers, explicit pool-to-tier mapping.
  • MTU 9000 matters. Jumbo frames on the storage VLAN produced roughly a 10× improvement in some small-block tests.
  • BlueFS DB/WAL offload turns HDDs into small-write contenders. HDDs with NVMe WAL hit 2,299 4 KB write IOPS — above the SATA SSD tier in the same test.
  • ZFS defaults can cost you an order of magnitude. Three settings took one TrueNAS pool from 375 to 4,271 MiB/s with zero hardware changes.
  • Verify the benchmark harness. A wrong grep string silently reported wrong IOPS for weeks.

Benchmark highlights

All Ceph numbers below are rados bench, 30 seconds per test, 16 threads, 3× replication, MTU 9000, measured on the production cluster in September 2026. TrueNAS numbers are fio runs on the four storage hosts.

1,175
MB/s · 4 MB NVMe read/write
852
MB/s · 4 MB HDD read
37,629
4 KB read IOPS · SSD
2,299
4 KB write IOPS · HDD + NVMe WAL
4,271
MiB/s · tuned ZFS sequential write
621K
4 KB random read IOPS · daghome

The surprise: HDDs beat SATA SSDs on small writes

HDDs with their WAL on NVMe reached 2,299 4 KB write IOPS versus 1,999 for the plain SATA SSD tier. The rotational write penalty largely disappears when the write-ahead log lives on flash. Reads still favor SSD (37,629 vs 23,988 IOPS), but the gap is smaller than intuition suggests.

See the full Ceph and TrueNAS benchmark tables, methodology, and caveats →

Field notes

Ceph benchmarks deep dive

September 20, 2026
What 99 OSDs across four storage classes actually deliver — and the benchmark parsing bug that nearly sent us down the wrong path.
Read the note →

How 3 ZFS settings cost us 11× performance

September 17, 2026
A 503 GiB TrueNAS host writing at 375 MiB/s. Three configuration changes later: 4,271 MiB/s. No hardware was replaced.
Read the note →

About the numbers

Snapshot, not live telemetry

The benchmark campaign was run in September 2026 against 14 nodes and 99 OSDs. The cluster has since grown to 15 nodes and 104 OSDs (531 TiB raw, 2,033 placement groups as of 20 November 2026). The benchmark results are published as measured, not extrapolated. Where a figure is current rather than historical, this site says so.

Maintained by GutmanLab. Content adapted from the lab's internal GutmanLabDocumentation repository and verified cluster notes. No credentials, internal addresses, or tenant data are published here.