What this site is
This is the public notebook for the compute and storage behind GutmanLab's pathology imaging work at Emory. It is part hardware tour, part operations log, and part benchmark archive. The cluster was not purchased as a single system; it accumulated. Each node has its own generation, RAM speed, drive mix, and personality.
Two clusters are documented here:
pmoxprime — the main cluster
kubertpmox — the DR cluster
ceph_nvme pool, CephFS with standby MDS, and backups to a
Proxmox Backup Server on the lab network.
Why "Frankencluster"
Most clusters are bought in one purchase order: identical nodes, identical drives, identical warranty dates. This one grew from hand-me-downs, surplus finds, grant leftovers, and temporary measures that became permanent. Three generations of server hardware share one Ceph cluster. RAM runs at 2133, 2400, and 2666 MT/s. Drive classes span 6 TB spinning disks to Intel Optane-accelerated SSDs.
The interesting engineering question is not "can heterogeneous hardware work?" It obviously can. The question is which workload should land on which tier, and how much performance a small configuration change can unlock. That is what the benchmark section is for.
The short version of what we learned
- CRUSH device classes are the abstraction that makes this sane. One cluster, four tiers, explicit pool-to-tier mapping.
- MTU 9000 matters. Jumbo frames on the storage VLAN produced roughly a 10× improvement in some small-block tests.
- BlueFS DB/WAL offload turns HDDs into small-write contenders. HDDs with NVMe WAL hit 2,299 4 KB write IOPS — above the SATA SSD tier in the same test.
- ZFS defaults can cost you an order of magnitude. Three settings took one TrueNAS pool from 375 to 4,271 MiB/s with zero hardware changes.
- Verify the benchmark harness. A wrong grep string silently reported wrong IOPS for weeks.
Benchmark highlights
All Ceph numbers below are rados bench, 30 seconds per test, 16 threads, 3× replication, MTU 9000, measured on the production cluster in September 2026. TrueNAS numbers are fio runs on the four storage hosts.
The surprise: HDDs beat SATA SSDs on small writes
HDDs with their WAL on NVMe reached 2,299 4 KB write IOPS versus 1,999 for the plain SATA SSD tier. The rotational write penalty largely disappears when the write-ahead log lives on flash. Reads still favor SSD (37,629 vs 23,988 IOPS), but the gap is smaller than intuition suggests.
See the full Ceph and TrueNAS benchmark tables, methodology, and caveats →
Field notes
Ceph benchmarks deep dive
How 3 ZFS settings cost us 11× performance
About the numbers
Snapshot, not live telemetry
The benchmark campaign was run in September 2026 against 14 nodes and 99 OSDs. The cluster has since grown to 15 nodes and 104 OSDs (531 TiB raw, 2,033 placement groups as of 20 November 2026). The benchmark results are published as measured, not extrapolated. Where a figure is current rather than historical, this site says so.
Maintained by GutmanLab. Content adapted from the lab's internal GutmanLabDocumentation repository and verified cluster notes. No credentials, internal addresses, or tenant data are published here.