On Sun, Nov 15, 2026 at 12:23:13PM +0400, Yasha Friend wrote:
Hi Richard,
I maintain hw-apex-bench (Apache-2.0), a small benchmark that measures the cost of reading one region of a compressed file — latency and decoded bytes — at different block sizes, for xz, zstd seekable and BGZF. Disclosure: I build it with heavy help from AI tools; every number is reproducible from the repository.
The nbdkit-xz-filter man page suggests many small blocks (16 MiB in its example) for random access. Would a table of region-read latency and ratio versus xz block size, measured on a public VM disk image, be useful to you or to users choosing --block-size? If so, which image and read sizes would you consider representative?
I'm not sure I completely understand the question, but I'll make a few hopefully relevant comments. As background, we have to recommend using --block-size because the default (of xz) is to put everything into a single block, which means that there's no way to random access the file without decoding everything up to that point in the file. With the --block-size parameter, only the block containing the data has to be decoded. Block size operates in the uncompressed space, but the LZMA file format includes a directory of block boundaries. There's a convenient API (lzma_index_iter_locate) to find the block boundary corresponding to an uncompressed offset using a binary search. We cache a few uncompressed blocks in nbdkit-xz-filter, with the number being controlled by xz-max-depth parameter. Choosing the block size to use during compression has to balance: * The impact on compression of using a smaller block size, since AIUI the dictionary cannot be used across block boundaries. * The amount of uncompressed data that has to be cached in memory, which is affected by the size of each block. * The time taken when reading a new block for the first time. A smaller block size makes compression worse, but reduces the size of the cache and the time taken to visit a new block. Is 16M a good size? Well, that's interesting, because I only measured the impact on compression (less than 1% worse). The amount of uncompressed memory is an obvious calculation from block size * xz-max-depth (16M * 8 = 128M). But I didn't measure the speed of the filter at different block sizes. So I guess my question is whether there is a better block size to recommend. Also the impact is limited since distros such as Fedora are using a block size of 16M already, and getting that changed is a slow process. Rich.
That is the whole question — "not useful" is a fine answer too.
Thanks, Yakiv Shavidze https://github.com/yasha1971-coder/hw-apex-bench
_______________________________________________ Libguestfs mailing list -- guestfs@lists.libguestfs.org To unsubscribe send an email to guestfs-leave@lists.libguestfs.org