Proxmox, briefly
Proxmox Virtual Environment (PVE) is a Debian-based virtualization platform. A PVE cluster is a group of physical hosts that share configuration, authentication, and a quorum service; each host can run QEMU virtual machines and LXC containers, and VMs can migrate between hosts.
Cluster and quorum
pvecm status shows membership and quorum.Storage, local or shared
Ceph inside Proxmox
Backups and HA
Why this matters for the tour
The cluster's flexibility comes from two layers: Proxmox decides where a VM runs, and Ceph decides where its blocks live. A VM's disk can be on a pool backed by HDDs, SATA SSDs, NVMe, or Optane-accelerated SSDs without changing anything inside the guest.
pmoxprime at a glance
pmoxprime is the production-adjacent cluster: fifteen Proxmox hosts, a seven-monitor Ceph quorum, and roughly half a petabyte of raw Ceph capacity. It is not a uniform deployment. Nodes differ in CPU generation, RAM population, drive count, and role.
Node hardware (September 2026 benchmark snapshot)
| Node | RAM | NVMe | SSD | HDD | Raw | Role |
|---|---|---|---|---|---|---|
| pmox2062 | 188 GiB | 3 | 1 | 11 | 83 TB | Storage workhorse |
| pmox2061 | 188 GiB | 2 | 1 | 10 | 75 TB | Heavy HDD host |
| titan | 188 GiB | 2 | 1 | 8 | 64 TB | Mixed storage |
| pmox2057 | 188 GiB | 1 | 1 | 7 | 45 TB | HDD bulk |
| pmox2059 | 377 GiB | 1 | 1 | 7 | 44 TB | HDD bulk |
| pmox2060 | 377 GiB | 1 | 1 | 6 | 44 TB | HDD bulk |
| pmox2058 | 314 GiB | 2 | 1 | 5 | 40 TB | Hybrid |
| pmox2055 | 503 GiB | 2 | 8 | 0 | 27 TB | All-flash compute |
| pmox2053 | 503 GiB | 1 | 7 | 0 | 25 TB | All-flash compute |
| pmox2064 | 503 GiB | 2 | 8 | 0 | 25 TB | All-flash + Optane |
| pmox2054 | 503 GiB | 2 | 7 | 0 | 23 TB | All-flash compute |
| pmox2056 | 503 GiB | 2 | 7 | 0 | 23 TB | All-flash compute |
| pmox2063 | 503 GiB | 0 | 8 | 0 | 21 TB | SSD-only |
| triton | 6 TB | 2 | 1 | 0 | 6 TB | Mon + mgr, control plane |
September 2026 snapshot. pmox2065 was added in November 2026 with three 10 TB HDDs and a local NVMe for WAL/DB; the cluster is now 15 nodes / 104 OSDs / 531 TiB raw. triton's 6 TB is flash used for the control plane, not bulk storage.
RAM and network
Total RAM is roughly 5.2 TiB of DDR4 ECC, a mix of 16 GB and 32 GB modules at 2133–2666 MT/s. Population is uneven by design: the all-flash compute nodes carry 503 GiB each, while the HDD hosts carry 188–377 GiB. Several nodes have empty DIMM slots, so memory can be expanded without replacing what is already there.
Storage traffic runs on a dedicated 25 GbE fabric (Mellanox ConnectX-4 Lx on the nodes that have been upgraded) with MTU 9000. A separate 10 GbE mesh carries public VM and management traffic. The Ceph public and cluster networks are separated; a misconfigured switch port or a single MTU 1500 host is enough to degrade the whole tier, which is why the fabric is checked end-to-end rather than per host.
Ceph: four tiers, one cluster
The architectural decision that makes heterogeneity work is the CRUSH device-class hierarchy. Each drive class gets its own rule, and each workload gets a pool that maps to the rule it needs. Ceph still treats the cluster as one pool of storage, but placement is explicit instead of accidental.
| Tier | OSDs* | Device class | CRUSH rule | Pools | Latency profile |
|---|---|---|---|---|---|
| HDD | 53 | hdd | replicated-hdd | ceph_bulk, cephfs_images_bulk (EC), profile_hdd | High capacity, moderate latency |
| SSD | 28 | ssd | replicated-ssd | ceph_ssd | Balanced |
| NVMe | 10 | nvme | replicated-nvme | ceph_nvme, profile_nvme | Low latency, high IOPS |
| Optane-backed | 8 | SSDoptane | ssdoptane_rule | ceph_optane | Ultra-low latency |
*Counts at the September 2026 benchmark. The cluster now has 104 OSDs after the pmox2065 HDD expansion.
BlueFS DB/WAL offload
Not every OSD keeps its data and metadata on the same device. HDD OSDs put their write-ahead log on NVMe; Optane-backed OSDs pair a 3.49 TB SATA SSD with a dedicated 64 GB NVMe partition for the BlueFS DB, with WAL co-located. This is the single biggest reason the HDD tier is competitive on small writes.
| Class | Data device | DB device | WAL device | Shared? |
|---|---|---|---|---|
| HDD | HDD | on HDD | NVMe | No |
| SSD | SATA SSD | on SSD | on SSD | Yes |
| NVMe | NVMe | on NVMe | on NVMe | Yes |
| Optane-backed | SATA SSD | 64 GB NVMe (dedicated) | co-located with DB | No |
Pool layout
| Pool | Type | Size | PGs | Rule | Apps |
|---|---|---|---|---|---|
| ceph_bulk | replicated | 3/2 | 512 | replicated-hdd | rbd |
| ceph_nvme | replicated | 3/2 | 128 | replicated-nvme | rbd |
| ceph_optane | replicated | 3/2 | 32 | ssdoptane_rule | rbd |
| ceph_ssd | replicated | 3/2 | 128 | replicated-ssd | rbd |
| cephfs_images_bulk | erasure-coded | 6/5 | 512 | cephfs_images_bulk | cephfs |
| cephfs_images_meta | replicated | 3/2 | 128 | replicated_rule | cephfs |
| profile_hdd / profile_nvme | replicated | 3/2 | 32 | hdd / nvme | rbd |
Current status (20 November 2026)
104/104 OSDs up and in, 2,033 placement groups active+clean, 7/7 monitors quorate, 531 TiB raw at roughly 48% used. The four device classes remain: HDD ~53% used, NVMe ~61%, SSD ~35%, Optane ~15%. The cluster is healthy; the remaining Ceph warnings are an auth-key migration item, not a data-path problem.
What runs on it
Alongside Ceph, four TrueNAS SCALE hosts provide NFS-backed bulk storage and the whole-slide imaging archive: brainhost/slidevault (36 HDDs, 458 TiB raw), icarus/daghome (all-SSD home and compute scratch), scratchy/samsungssd (all-SSD scratch), and gauss/tank (HDD bulk). Two QNAP units hold archival WSI data and restic targets. The benchmark story for those hosts is on the benchmarks page.
A separate cluster, kubertpmox, handles disaster recovery and staging so that backups are not trapped inside the system they protect.