[perf] NVMe randwrite: 2.4.0 slower on dataset/zvol vs 2.2.4 — please help validate test methodology (Direct IO/ARC)

[pickup112]

Background

• I noticed the 2.3.0 release notes mention Direct I/O (Direct IO Support #10018): bypassing ARC for reads/writes, which should help in NVMe scenarios.
• My workload is database-heavy on ZFS, so random write I/O is important. I compared 2.2.4 vs 2.4.0.
• The results are not what I expected: 2.4.0 shows lower IOPS/BW and higher tail latency on dataset and zvol. I’d like to ask the community to first
  validate whether my test methodology is reasonable.

Environment

• Host: 10.10.150.235
• OS: Kylin Linux V10, kernel 4.19.90-89.11.v2401.ky10.x86_64
• ZFS: zfs-2.4.0/zfs-kmod-2.4.0 vs zfs-2.2.4/zfs-kmod-2.2.4
• Device: NVMe P6F3840 (/dev/nvme7n1)
• Test targets: raw device, zvol block device, ZFS dataset

Methodology

• fio parameters:

ioengine=libaio
direct=1
rw=randwrite
bs=8k
numjobs=4
iodepth=8
runtime=120

• dataset: nrfiles=10, filesize=100g
• raw/zvol: size=1t
• Each test repeated 3 times, averaged

ZFS configuration

• pool: listsnapshots=on, ashift=12, atime=off, primarycache=metadata
• dataset: recordsize=32k, compression=off, dedup=off
• zvol: volblocksize=32k, compression=off, dedup=off

Results (average of 3 runs)

Scenario	2.2.4 IOPS	2.4.0 IOPS	ΔIOPS	2.2.4 BW (MiB/s)	2.4.0 BW (MiB/s)	ΔBW	2.2.4 p99 (ms)	2.4.0 p99 (ms)	Δp99	2.2.4 p99.9 (ms)	2.4.0 p99.9 (ms)	Δp99.9
raw	174711	155139	-11.2%	1364.9	1212.0	-11.2%	0.923	0.744	-19.4%	1.059	1.470	+38.8%
dataset	22467	20342	-9.5%	175.5	158.9	-9.5%	8.094	20.152	+149.0%	15.750	49.152	+212.1%
zvol	27588	18549	-32.8%	215.5	144.9	-32.8%	16.417	25.996	+58.3%	47.142	46.749	-0.8%

Questions / feedback requested

• The docs say Direct I/O writes need recordsize alignment. My test uses bs=8k with recordsize=32k. Could this cause the writes to fall back to ARC? Should
  I adjust to recordsize=8k or bs=32k to properly exercise Direct I/O?
• The docs mention Direct I/O is not supported for zvols. Does that mean zvol performance should not be expected to benefit from the Direct I/O change?
• If there is a better fio job or ZFS tunable set to validate NVMe Direct I/O improvements, please let me know (I can provide full jobfiles/commands/raw
  outputs).

[amotin]

Random 8KB write with record-/volblocksize of 32KB requite read-modify-write on every request with predictable performance results. Enabled ZFS caching is the only way to avoid it, but if the active data set is too big, then it won't help much. Obviously you can't do Direct I/O in that situation, as data has to be aggregated somewhere.

Direct I/O indeed is not implemented for ZVOLs. Passing O_DIRECT there will make Linux to bypass its page cache, which is not the same.

Direct I/O tries to save on memory copies. Doing 8KB I/Os the memory copies is the least of your bottlenecks. The first you must do is to avoid read-modify-write by making application writes multiple and aligned to record-/volblocksize. After that you may experiment with either Uncached I/O (primarycache=metadata) or Direct I/O (O_DIRECT).

[pickup112]

Thanks for the suggestion! My ultimate goal is to find the best practices for ZFS on NVMe. My use case is Oracle, which defaults to 8K blocks, so I will focus on testing with recordsize=8K and, on 2.4.0, explicitly set direct=always to verify the real benefit of Direct I/O. I will also use the same test method to compare 2.2.4 with 2.4.0, and I’ll update this thread with the full test conditions and results.

[pickup112]

I re-ran on 10.10.150.235 , single‑disk pool (ashift=12) and the same fio job from this discussion (8k randwrite; 120s; warmup + 3 runs; median; drop caches between runs). The disk and layout are essentially the same; the main difference is recordsize=8K

Dataset settings: recordsize=8K , atime=off, compression=off, dedup=off, primarycache=metadata.

Results (median IOPS / BW / clat mean / p99):

• Raw NVMe: 183k / 1432 MiB/s / 170us / 651us
• ZFS 2.2.4, recordsize=8K: 59k / 464 MiB/s / 472us / 1.40ms
• ZFS 2.4.0, recordsize=8K, direct=always: 27k / 213 MiB/s / 1.03ms / 4.36ms
• ZFS 2.4.0, recordsize=8K, direct=standard: 27k / 215 MiB/s / 1.02ms / 4.36ms
• ZFS 2.4.0, recordsize=8K, direct=disabled: 64k / 502 MiB/s / 437us / 1.42ms

he best result is 2.4.0 with direct=disabled (slightly better than 2.2.4). direct=always/standard is much worse in this workload.

Could you help confirm why the O_DIRECT improvement in 2.4 doesn’t show up here? Is there a specific alignment/IO pattern requirement, or any tunables/flags needed for the direct property to be effective on datasets?

Thanks! I can share full fio outputs if helpful.

[amotin]

O_DIRECT is a very special beast. It requires your workload to have a really deep request queue depth (both in requests and megabytes) to cover the latency. And the one you have might be not enough. Also you are comparing it with primarycache=metadata, which already reduces ZFS caching to as little as reasonable, and attempt to get more by force may be counterproductive. As I have told before, at 8KB I/Os memory bandwidth saved by O_DIRECT likely does not worth losing write caching and other downsides.

[pickup112]

Thanks for the detailed explanation. My understanding is that under my current test conditions (primarycache=metadata, fio iodepth=8, bs=8K), the O_DIRECT optimizations introduced after 2.3 are unlikely to be visible. In particular, if the zvol block size is 32K while fio writes are 8K, that can trigger RMW; and since zvol does not use a true direct I/O path, any optimization impact can be further masked.

I’ll run another comparison in a scenario closer to the intended one: set primarycache=all, align block sizes to 32K (recordsize and fio bs all set to 32K), and increase iodepth to 128. Under the same hardware and kernel conditions, I’ll compare IOPS, throughput, and latency between 2.2.4 and 2.4.0. If my understanding of the applicable scenarios is still off, please feel free to correct me.