cluster.drgutman.org

Proxmox, briefly

Proxmox Virtual Environment (PVE) is a Debian-based virtualization platform. A PVE cluster is a group of physical hosts that share configuration, authentication, and a quorum service; each host can run QEMU virtual machines and LXC containers, and VMs can migrate between hosts.

Cluster and quorum

Nodes join a cluster through Corosync. A majority of votes must be present for the cluster to make changes. pvecm status shows membership and quorum.

Storage, local or shared

PVE can use local disks (LVM, ZFS, directories) or shared storage such as NFS, iSCSI, or Ceph RBD. Shared storage is what makes live migration and HA practical.

Ceph inside Proxmox

PVE can deploy and manage Ceph directly. Each host contributes OSDs; pools are exposed to PVE as storage. Replication and placement are handled by Ceph, not by RAID.

Backups and HA

Proxmox Backup Server provides incremental, deduplicated backups. HA moves or restarts guests when a node fails, but only for resources explicitly placed under HA management.

Why this matters for the tour

The cluster's flexibility comes from two layers: Proxmox decides where a VM runs, and Ceph decides where its blocks live. A VM's disk can be on a pool backed by HDDs, SATA SSDs, NVMe, or Optane-accelerated SSDs without changing anything inside the guest.

pmoxprime at a glance

pmoxprime is the production-adjacent cluster: fifteen Proxmox hosts, a seven-monitor Ceph quorum, and roughly half a petabyte of raw Ceph capacity. It is not a uniform deployment. Nodes differ in CPU generation, RAM population, drive count, and role.

pmoxprime topology
┌──────────────────────────────────────────────────────────┐ │ 15 × Proxmox nodes │ │ pmox2053–pmox2065, titan, triton │ │ 25 GbE Ceph fabric (MTU 9000) + 10 GbE public mesh │ ├──────────────────────────────────────────────────────────┤ │ Ceph Squid │ │ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────────┐ │ │ │ HDD │ │ SSD │ │ NVMe │ │ Optane- │ │ │ │ │ │ │ │ │ │ backed │ │ │ └──────┘ └──────┘ └──────┘ └──────────┘ │ │ │ │ │ │ │ │ replicated replicated replicated ssdoptane_rule │ │ -hdd -ssd -nvme │ ├──────────────────────────────────────────────────────────┤ │ 4 × TrueNAS SCALE hosts · 2 × QNAP archival NAS │ │ Proxmox Backup Server · separate kubertpmox DR cluster │ └──────────────────────────────────────────────────────────┘

Node hardware (September 2026 benchmark snapshot)

NodeRAMNVMeSSDHDDRawRole
pmox2062188 GiB311183 TBStorage workhorse
pmox2061188 GiB211075 TBHeavy HDD host
titan188 GiB21864 TBMixed storage
pmox2057188 GiB11745 TBHDD bulk
pmox2059377 GiB11744 TBHDD bulk
pmox2060377 GiB11644 TBHDD bulk
pmox2058314 GiB21540 TBHybrid
pmox2055503 GiB28027 TBAll-flash compute
pmox2053503 GiB17025 TBAll-flash compute
pmox2064503 GiB28025 TBAll-flash + Optane
pmox2054503 GiB27023 TBAll-flash compute
pmox2056503 GiB27023 TBAll-flash compute
pmox2063503 GiB08021 TBSSD-only
triton6 TB2106 TBMon + mgr, control plane

September 2026 snapshot. pmox2065 was added in November 2026 with three 10 TB HDDs and a local NVMe for WAL/DB; the cluster is now 15 nodes / 104 OSDs / 531 TiB raw. triton's 6 TB is flash used for the control plane, not bulk storage.

RAM and network

Total RAM is roughly 5.2 TiB of DDR4 ECC, a mix of 16 GB and 32 GB modules at 2133–2666 MT/s. Population is uneven by design: the all-flash compute nodes carry 503 GiB each, while the HDD hosts carry 188–377 GiB. Several nodes have empty DIMM slots, so memory can be expanded without replacing what is already there.

Storage traffic runs on a dedicated 25 GbE fabric (Mellanox ConnectX-4 Lx on the nodes that have been upgraded) with MTU 9000. A separate 10 GbE mesh carries public VM and management traffic. The Ceph public and cluster networks are separated; a misconfigured switch port or a single MTU 1500 host is enough to degrade the whole tier, which is why the fabric is checked end-to-end rather than per host.

Ceph: four tiers, one cluster

The architectural decision that makes heterogeneity work is the CRUSH device-class hierarchy. Each drive class gets its own rule, and each workload gets a pool that maps to the rule it needs. Ceph still treats the cluster as one pool of storage, but placement is explicit instead of accidental.

TierOSDs*Device classCRUSH rulePoolsLatency profile
HDD53hddreplicated-hddceph_bulk, cephfs_images_bulk (EC), profile_hddHigh capacity, moderate latency
SSD28ssdreplicated-ssdceph_ssdBalanced
NVMe10nvmereplicated-nvmeceph_nvme, profile_nvmeLow latency, high IOPS
Optane-backed8SSDoptanessdoptane_ruleceph_optaneUltra-low latency

*Counts at the September 2026 benchmark. The cluster now has 104 OSDs after the pmox2065 HDD expansion.

BlueFS DB/WAL offload

Not every OSD keeps its data and metadata on the same device. HDD OSDs put their write-ahead log on NVMe; Optane-backed OSDs pair a 3.49 TB SATA SSD with a dedicated 64 GB NVMe partition for the BlueFS DB, with WAL co-located. This is the single biggest reason the HDD tier is competitive on small writes.

ClassData deviceDB deviceWAL deviceShared?
HDDHDDon HDDNVMeNo
SSDSATA SSDon SSDon SSDYes
NVMeNVMeon NVMeon NVMeYes
Optane-backedSATA SSD64 GB NVMe (dedicated)co-located with DBNo

Pool layout

PoolTypeSizePGsRuleApps
ceph_bulkreplicated3/2512replicated-hddrbd
ceph_nvmereplicated3/2128replicated-nvmerbd
ceph_optanereplicated3/232ssdoptane_rulerbd
ceph_ssdreplicated3/2128replicated-ssdrbd
cephfs_images_bulkerasure-coded6/5512cephfs_images_bulkcephfs
cephfs_images_metareplicated3/2128replicated_rulecephfs
profile_hdd / profile_nvmereplicated3/232hdd / nvmerbd

Current status (20 November 2026)

104/104 OSDs up and in, 2,033 placement groups active+clean, 7/7 monitors quorate, 531 TiB raw at roughly 48% used. The four device classes remain: HDD ~53% used, NVMe ~61%, SSD ~35%, Optane ~15%. The cluster is healthy; the remaining Ceph warnings are an auth-key migration item, not a data-path problem.

What runs on it

Alongside Ceph, four TrueNAS SCALE hosts provide NFS-backed bulk storage and the whole-slide imaging archive: brainhost/slidevault (36 HDDs, 458 TiB raw), icarus/daghome (all-SSD home and compute scratch), scratchy/samsungssd (all-SSD scratch), and gauss/tank (HDD bulk). Two QNAP units hold archival WSI data and restic targets. The benchmark story for those hosts is on the benchmarks page.

A separate cluster, kubertpmox, handles disaster recovery and staging so that backups are not trapped inside the system they protect.