Question: xz block size vs region-read cost
Hi Richard, I maintain hw-apex-bench (Apache-2.0), a small benchmark that measures the cost of reading one region of a compressed file — latency and decoded bytes — at different block sizes, for xz, zstd seekable and BGZF. Disclosure: I build it with heavy help from AI tools; every number is reproducible from the repository. The nbdkit-xz-filter man page suggests many small blocks (16 MiB in its example) for random access. Would a table of region-read latency and ratio versus xz block size, measured on a public VM disk image, be useful to you or to users choosing --block-size? If so, which image and read sizes would you consider representative? That is the whole question — "not useful" is a fine answer too. Thanks, Yakiv Shavidze https://github.com/yasha1971-coder/hw-apex-bench
On Sun, Nov 15, 2026 at 12:23:13PM +0400, Yasha Friend wrote:
Hi Richard,
I maintain hw-apex-bench (Apache-2.0), a small benchmark that measures the cost of reading one region of a compressed file — latency and decoded bytes — at different block sizes, for xz, zstd seekable and BGZF. Disclosure: I build it with heavy help from AI tools; every number is reproducible from the repository.
The nbdkit-xz-filter man page suggests many small blocks (16 MiB in its example) for random access. Would a table of region-read latency and ratio versus xz block size, measured on a public VM disk image, be useful to you or to users choosing --block-size? If so, which image and read sizes would you consider representative?
I'm not sure I completely understand the question, but I'll make a few hopefully relevant comments. As background, we have to recommend using --block-size because the default (of xz) is to put everything into a single block, which means that there's no way to random access the file without decoding everything up to that point in the file. With the --block-size parameter, only the block containing the data has to be decoded. Block size operates in the uncompressed space, but the LZMA file format includes a directory of block boundaries. There's a convenient API (lzma_index_iter_locate) to find the block boundary corresponding to an uncompressed offset using a binary search. We cache a few uncompressed blocks in nbdkit-xz-filter, with the number being controlled by xz-max-depth parameter. Choosing the block size to use during compression has to balance: * The impact on compression of using a smaller block size, since AIUI the dictionary cannot be used across block boundaries. * The amount of uncompressed data that has to be cached in memory, which is affected by the size of each block. * The time taken when reading a new block for the first time. A smaller block size makes compression worse, but reduces the size of the cache and the time taken to visit a new block. Is 16M a good size? Well, that's interesting, because I only measured the impact on compression (less than 1% worse). The amount of uncompressed memory is an obvious calculation from block size * xz-max-depth (16M * 8 = 128M). But I didn't measure the speed of the filter at different block sizes. So I guess my question is whether there is a better block size to recommend. Also the impact is limited since distros such as Fedora are using a block size of 16M already, and getting that changed is a slow process. Rich.
That is the whole question — "not useful" is a fine answer too.
Thanks, Yakiv Shavidze https://github.com/yasha1971-coder/hw-apex-bench
_______________________________________________ Libguestfs mailing list -- guestfs@lists.libguestfs.org To unsubscribe send an email to guestfs-leave@lists.libguestfs.org
Hi Rich, Thanks for the explanation. I measured the time to read a block for the first time through nbdkit-xz-filter, at different block sizes. Setup: Fedora-Cloud-Base-AmazonEC2-45-1.4 raw (sha256 from the official CHECKSUM), recompressed with xz -6 at each block size. The 24M archive is byte-identical to Fedora's own .raw.xz: 24 MiB is simply xz's default block size in multi-threaded mode (3 x the 8 MiB dictionary of -6), so that is what Fedora ships, not 16M. nbdkit 1.24.1 (Ubuntu 22.04), liblzma 5.2.5, xz-max-depth default, libnbd client, 4 KiB reads at random offsets inside blocks that contain data; 1,000 reads per point, median of 5 runs, every reply checked byte-for-byte against the raw image. block size vs 24M first read p50 / p99 cached read nbdkit peak RSS 1M +3.1% 27 / 34 ms 0.024 ms 16 MiB 2M +2.2% 52 / 69 ms 0.023 ms 26 MiB 4M +1.5% 110 / 137 ms 0.024 ms 48 MiB 8M +0.8% 144 / 272 ms 0.025 ms 116 MiB 16M +0.4% 224 / 545 ms 0.026 ms 157 MiB 24M 0 276 / 804 ms 0.028 ms 253 MiB 64M -0.3% 597 / 2128 ms 0.029 ms 589 MiB (plain nbdkit file on the raw image: 0.014 ms per 4 KiB read) The first-read cost grows with block size, but not linearly, and even 1M costs 27 ms. This is an old nbdkit version, so please treat the absolute numbers with that caveat. So there is no single best size, but going from 24M to 1-4M cuts first-read latency by 2.5-10x for +1.5-3.1% size and much less memory. Whether that matters depends on how often a VM touches a new block. Raw data, scripts and the full table (including uniform reads over the whole image, where zero blocks make reads cheaper): https://www.google.com/url?q=https://github.com/yasha1971-coder/hw-apex-bench/blob/3c9a68329ba591603f20a7405af0de853ee09c67/evidence/hwab-nbd-1-20261121/README.md&source=gmail&ust=1795376551358000&sa=E Yakiv On Sun, Nov 15, 2026 10:23 AM, Yasha Friend <yasha1971@gmail.com> wrote:
Hi Richard,
I maintain hw-apex-bench (Apache-2.0), a small benchmark that measures the cost of reading one region of a compressed file — latency and decoded bytes — at different block sizes, for xz, zstd seekable and BGZF. Disclosure: I build it with heavy help from AI tools; every number is reproducible from the repository.
The nbdkit-xz-filter man page suggests many small blocks (16 MiB in its example) for random access. Would a table of region-read latency and ratio versus xz block size, measured on a public VM disk image, be useful to you or to users choosing --block-size? If so, which image and read sizes would you consider representative?
That is the whole question — "not useful" is a fine answer too.
Thanks,
Yakiv Shavidze
participants (2)
-
Richard W.M. Jones -
Yasha Friend