Re: [Libguestfs] [PATCH nbdkit INCOMPLETE] readahead: Rewrite this filter so it prefetches using .cache

On Tue, Apr 19, 2022 at 04:31:44PM +0200, Laszlo Ersek wrote: > On 04/19/22 14:47, Richard W.M. Jones wrote: > > The previous readahead filter did not work well and we have stopped > > using it in virt-v2v. However the concept is a good one if we can > > make it work. This commit completely rethinks the filter. > > I started responding to the cover letter, "what is the motivation for > this? furthermore, can we add this as a new filter instead, so > comparisons (back-and-forth) can be made easily"? TBH the motivation is not very clear, but here are several current reasons: (1) Virt-v2v does some random access conversion up front, but then spends most of its time sequentially copying, using originally “qemu-img convert” and now nbdcopy. It was thought that nbdkit-readahead-filter could help in some way, but the current implementation does not help, and in fact had some pretty negative effects. If we believe that nbdkit can help (which is not entirely clear), then fixing nbdkit-readahead-filter is worthwhile. For virt-v2v it's often the case that the source is remote, over a relatively slow network. On the one hand, prefetching would be good because it gets the data across the network and into the local cache. On the other hand, it's likely we can saturate most networks, so prefetching will make no difference or be net negative.

(2) Libvirt XML exposes the "readahead" setting of the qemu curl driver. https://libvirt.org/formatdomain.html#hard-drives-floppy-disks-cdroms https://gitlab.com/qemu/qemu/-/blob/master/block/curl.c#L897 In order to convert libvirt to use nbdkit-curl-plugin it's therefore felt that we ought to have an equivalent feature in nbdkit, although another way to fix this would simply be to ignore this setting.

(3) We have a readahead filter which definitely doesn't work, so we ought to either fix it or retire it. (4) A feature of nbdkit is that you can write simple plugins (eg. no threads, no fancy features), and then layer filters on top to add some of those features (caching, logging, readahead), so that would be a reason to have a readahead filter.