How 3 ZFS Settings Cost Us 11× Performance
A 503 GiB machine with 8 SSDs in mirror + NVMe special vdev + SLOG was writing at 375 MiB/s. After three configuration changes and zero hardware swaps: 4,271 MiB/s.
The Setup
Our TrueNAS SCALE host named Icarus serves the
daghome pool — home directories, compute scratch, and shared
project data for a pathology research lab. The hardware looked great on
paper:
- 8 × SATA SSD in 4 mirror vdevs
- 2 × NVMe mirror as a special (metadata + small-block) vdev
- 1 × 120 GB Intel NVMe as a dedicated SLOG
- 503 GiB DDR4 ECC RAM
- Dual 25 GbE networking, MTU 9000
This should have been a beast. Sequential writes should have been in the
multiple-GB/s range. Instead, routine rsync jobs crawled,
NFS felt sluggish, and a simple fio sequential write benchmark confirmed
the bad news: 375 MiB/s.
That's about what a single SATA SSD can do, or one half of a 10 GbE link. Something was very wrong.
The Investigation
We ran a full fio suite and started peeling back layers. The
hardware wasn't faulty — iostat showed the 8 SATA SSDs
mostly idle during writes. The bottleneck was software.
Bottleneck #1: special_small_blocks=128K + recordsize=16K
TrueNAS's special vdev is meant for metadata and small-file data. The
special_small_blocks tunable sets the size threshold —
anything at or below it goes to the special vdev instead of the main pool.
We had it set to 128 KB.
Our dataset's recordsize was 16 KB. Every
file block (16 KB) was under the 128 KB threshold, so every
single write was diverted to the 2× NVMe special mirror. The 8×
SATA SSDs sat idle for writes. The special vdev — meant to accelerate
metadata — became the main write path and the bottleneck.
special_small_blocks=0. Now
metadata-only goes to NVMe, and data spreads across the main pool's 8
SSDs as intended.
Result: 375 → 478 MiB/s (+27%)
Bottleneck #2: logbias=latency
The logbias property controls whether ZFS writes
synchronous data only (the default interpretation) or all
data through the ZIL/SLOG. logbias=latency forces every write
— even asynchronous writes that should be aggregated in RAM and flushed
as large chunks — through the SLOG.
Our SLOG was a single 120 GB Intel NVMe. Its sequential write throughput
tops out around 360 MB/s. With logbias=latency, every write
had to pass through that single device serially, creating a hard
~360 MB/s ceiling regardless of how fast the main pool was.
This setting is appropriate for NFS exports of database storage where every write absolutely must survive a power loss. For home directories, NFS file serving, and compute scratch, it's pure overhead.
logbias=throughput. Async writes
now bypass the ZIL entirely and aggregate into large transactions in RAM.
Result: 375 → 524 MiB/s (+40% cumulative)
Bottleneck #3: zfs_dirty_data_max=4 GB
ZFS accumulates dirty (unwritten) data in RAM and flushes it in large
transaction groups. The zfs_dirty_data_max tunable caps how
much dirty data is allowed before the transaction group throttle kicks in
and starts blocking writes.
Traditionally, OpenZFS defaults this to 10% of RAM (about 50 GB on a 503 GiB machine). But TrueNAS SCALE ships with it hard-coded to 4 GB — a paltry 0.8%. Why? Because TrueNAS used to run on appliances with 8-16 GB of RAM where 10% made sense. SCALE inherited that default, and nobody changed it for large-memory deployments.
With 503 GiB of RAM and a 4 GB cap, ZFS was constantly hitting its throttle. Dirty data would accumulate in 5-10 seconds, block new writes until the TXG flushed, then repeat. Bulk write throughput was limited not by the SSDs or the network, but by ZFS's own write-throttling mechanism.
zfs_dirty_data_max=32 GB (about 6%
of RAM). For write-heavy workloads, 10-20% would be fine.
Result: 524 → 4,271 MiB/s (+1,039% cumulative)
The Results
| Configuration | Sequential Write | vs. Baseline |
|---|---|---|
| Original (all 3 bottlenecks) | 375 MiB/s | — |
After special_small_blocks=0 |
478 MiB/s | +27% |
After logbias=throughput |
524 MiB/s | +40% |
After dirty_data=8 GB |
568 MiB/s | +51% |
After dirty_data=32 GB |
4,271 MiB/s | +1,039% |
That's an 11× improvement from three configuration changes. No hardware was swapped. No drivers were updated. No cables were reseated. Just ZFS parameters that had been inadvertently misconfigured by a combination of defaults and well-intentioned but wrong settings.
We also benchmarked with psync (NFS-like sync writes) to
simulate real-world workload patterns. The optimized pool hit
1,549 MiB/s — enough to saturate a 25 GbE link.
Random Read IOPS
After the fix, 4K random reads hit 621,000 IOPS at 2,424 MiB/s, and mixed 70/30 read/write sustained 54,800 read + 23,500 write IOPS. The pool is now extremely capable for both bulk throughput and metadata-heavy workloads.
Key Takeaways
-
special_small_blocksis powerful but dangerous. Combined with a smallrecordsize, it can accidentally turn your metadata accelerator into your main write path and biggest bottleneck. Set it to 0 unless you know exactly what you're doing and have tested the workload. -
logbias=latencyis for databases, not file servers. Use it only when every write must survive a power loss. For NFS home directories, SMB shares, and compute scratch,throughputis almost always correct. - TrueNAS SCALE ships with zfs_dirty_data_max=4 GB. On machines with more than 64 GB of RAM, this is insanely conservative. Raise it to at least 10% of RAM for write-heavy workloads. On our 503 GiB machine, going from 4 GB → 32 GB was the single biggest performance lever.
-
Benchmark with the right access pattern.
direct=1(bypasses ARC/page cache) andpsync(simulates NFS behavior) can give wildly different results. Test your actual workload, not a synthetic best-case. - Verify your defaults. TrueNAS makes reasonable assumptions for embedded appliances. On server-class hardware, many of those assumptions are wrong. Profile before you assume it's "fine."
How to Check Your Own System
Run these on your TrueNAS or vanilla ZFS system:
# Check current settings
zfs get special_small_blocks
zfs get logbias
echo "zfs_dirty_data_max = $(cat /sys/module/zfs/parameters/zfs_dirty_data_max)"
# Permanent fix (on TrueNAS SCALE, add to System → Tunables):
# special_small_blocks=0
# zfs_dirty_data_max=34359738368 (32 GB — adjust for your RAM)
The logbias change is a dataset property, so apply it to
individual datasets. Home directories and general-purpose shares should
use throughput. Databases and iSCSI LUNs can stay on
latency.
What's Next
We're running similar tuning on our other TrueNAS hosts (brainhost,
gauss, scratchy). Early results suggest the zfs_dirty_data_max
fix helps all of them, while special_small_blocks and
logbias corrections are dataset-specific.
For more on our infrastructure, see the full Frankencluster write-up, which covers the Proxmox+Ceph cluster, all four TrueNAS hosts, and comprehensive benchmark results across every storage tier.
Originally published in the GutmanLab internal documentation. Benchmarks measured September 2026; see the benchmark index for current caveats.