[PATCH nbdkit] v2v: Disable readahead for VMware curl sources too (RHBZ#1848862).
I'm still testing this fix, so let's hold off the review for the moment. Also it may be better to specifically identify problematic servers rather than disabling this for every curl source. eg. I suspect that the problem is the Java server used by VCenter, so we might think about only disabling readahead for that single case. Rich.
This appears to be the cause of timeouts during the conversion step where VMware VCenter server's Tomcat HTTPS server stops responding to requests (or rather, responds only with 503 errors). The server later recovers and in fact because of the retry filter the conversion usually succeeds, but I found that we can avoid the problem by disabling readahead. --- v2v/nbdkit_sources.ml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/v2v/nbdkit_sources.ml b/v2v/nbdkit_sources.ml index f5e919116..7c177e358 100644 --- a/v2v/nbdkit_sources.ml +++ b/v2v/nbdkit_sources.ml @@ -99,12 +99,12 @@ let common_create ?bandwidth ?extra_debug ?extra_env password (* Adding the readahead filter is always a win for our access * patterns. If it doesn't exist don't worry. However it - * breaks VMware servers (RHBZ#1832805). + * breaks VMware servers (RHBZ#1832805, RHBZ#1848862). *) let cmd = - if plugin_name <> "vddk" then - Nbdkit.add_filter_if_available cmd "readahead" - else cmd in + match plugin_name with + | "vddk" | "curl" -> cmd + | _ -> Nbdkit.add_filter_if_available cmd "readahead" in (* Caching extents speeds up qemu-img, especially its consecutive * block_status requests with req_one=1. -- 2.25.0
On 6/19/20 7:47 AM, Richard W.M. Jones wrote:
This appears to be the cause of timeouts during the conversion step where VMware VCenter server's Tomcat HTTPS server stops responding to requests (or rather, responds only with 503 errors). The server later recovers and in fact because of the retry filter the conversion usually succeeds, but I found that we can avoid the problem by disabling readahead. --- v2v/nbdkit_sources.ml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
Is it just a matter of readahead requesting too much data at once? Is it time to start revisiting the idea of moving the core of the 'blocksize' filter into the server itself, in order to make it easier for various plugins and filters to request min/max block limits, and where the common server code then stitches together read-modify-write or splits requests as needed to the next layer? -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Fri, Jun 19, 2020 at 10:13:05AM -0500, Eric Blake wrote:
On 6/19/20 7:47 AM, Richard W.M. Jones wrote:
This appears to be the cause of timeouts during the conversion step where VMware VCenter server's Tomcat HTTPS server stops responding to requests (or rather, responds only with 503 errors). The server later recovers and in fact because of the retry filter the conversion usually succeeds, but I found that we can avoid the problem by disabling readahead. --- v2v/nbdkit_sources.ml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
Is it just a matter of readahead requesting too much data at once? Is it time to start revisiting the idea of moving the core of the 'blocksize' filter into the server itself, in order to make it easier for various plugins and filters to request min/max block limits, and where the common server code then stitches together read-modify-write or splits requests as needed to the next layer?
In fact this patch doesn't seem to have fixed the problem (it seemed to fix it for me, but not for mxie). However I do believe that our readahead filter is very naive and could do with a rewrite, but blocksize in the server may not help. In particular: * virt-v2v <= 1.40 (when using qemu's curl driver) used to adjust the qemu curl driver readahead setting between the conversion and copying steps, essentially only applying readahead when copying. Conversion has somewhat random access patterns, whereas copying is mostly linear. * We enable qemu's copyonread feature to cache reads in the overlay. However this certainly interacts badly with nbdkit-readahead-filter for a couple of reasons: (1) copyonread breaks linear reads, thus defeating the filter heuristics. (2) the readahead filter will read beyond the requested region, but it doesn't "know" if that data was already cached in the overlay by a previous read, therefore it reads more than necessary. (It's also possible - but I didn't check - that these poor interactions don't happen with qemu's curl driver readahead + copyonread. At least, it's all running in the same binary.) * VMware is a piece of crap and can't handle large reads without crashing (neither over HTTPS nor VDDK as it happens). Here, breaking up large reads could help, but we wouldn't want to bake that into nbdkit curl plugin because it's only applicable when using VMware's failing software, most HTTP servers are fine. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
participants (2)
-
Eric Blake -
Richard W.M. Jones