Re: [Libguestfs] [PATCH nbdkit] v2v: Disable readahead for VMware curl sources too (RHBZ#1848862).
On Fri, Jun 19, 2020 at 10:13:05AM -0500, Eric Blake wrote:
On 6/19/20 7:47 AM, Richard W.M. Jones wrote:
>This appears to be the cause of timeouts during the conversion step
>where VMware VCenter server's Tomcat HTTPS server stops responding to
>requests (or rather, responds only with 503 errors). The server later
>recovers and in fact because of the retry filter the conversion
>usually succeeds, but I found that we can avoid the problem by
>disabling readahead.
>---
> v2v/nbdkit_sources.ml | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
Is it just a matter of readahead requesting too much data at once?
Is it time to start revisiting the idea of moving the core of the
'blocksize' filter into the server itself, in order to make it
easier for various plugins and filters to request min/max block
limits, and where the common server code then stitches together
read-modify-write or splits requests as needed to the next layer?
In fact this patch doesn't seem to have fixed the problem (it seemed
to fix it for me, but not for mxie).
However I do believe that our readahead filter is very naive and could
do with a rewrite, but blocksize in the server may not help. In
particular:
* virt-v2v <= 1.40 (when using qemu's curl driver) used to adjust the
qemu curl driver readahead setting between the conversion and
copying steps, essentially only applying readahead when copying.
Conversion has somewhat random access patterns, whereas copying is
mostly linear.
* We enable qemu's copyonread feature to cache reads in the overlay.
However this certainly interacts badly with nbdkit-readahead-filter
for a couple of reasons: (1) copyonread breaks linear reads, thus
defeating the filter heuristics. (2) the readahead filter will read
beyond the requested region, but it doesn't "know" if that data was
already cached in the overlay by a previous read, therefore it reads
more than necessary.
(It's also possible - but I didn't check - that these poor
interactions don't happen with qemu's curl driver readahead +
copyonread. At least, it's all running in the same binary.)
* VMware is a piece of crap and can't handle large reads without
crashing (neither over HTTPS nor VDDK as it happens). Here,
breaking up large reads could help, but we wouldn't want to bake
that into nbdkit curl plugin because it's only applicable when using
VMware's failing software, most HTTP servers are fine.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/