7:47 a.m.
I'm still testing this fix, so let's hold off the review for the
moment.
Also it may be better to specifically identify problematic servers
rather than disabling this for every curl source. eg. I suspect that
the problem is the Java server used by VCenter, so we might think
about only disabling readahead for that single case.
Rich.
7:47 a.m.
This appears to be the cause of timeouts during the conversion step
where VMware VCenter server's Tomcat HTTPS server stops responding to
requests (or rather, responds only with 503 errors). The server later
recovers and in fact because of the retry filter the conversion
usually succeeds, but I found that we can avoid the problem by
disabling readahead.
---
v2v/nbdkit_sources.ml | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/v2v/nbdkit_sources.ml b/v2v/nbdkit_sources.ml
index f5e919116..7c177e358 100644
--- a/v2v/nbdkit_sources.ml
+++ b/v2v/nbdkit_sources.ml
@@ -99,12 +99,12 @@ let common_create ?bandwidth ?extra_debug ?extra_env password
(* Adding the readahead filter is always a win for our access
* patterns. If it doesn't exist don't worry. However it
- * breaks VMware servers (RHBZ#1832805).
+ * breaks VMware servers (RHBZ#1832805, RHBZ#1848862).
*)
let cmd =
- if plugin_name <> "vddk" then
- Nbdkit.add_filter_if_available cmd "readahead"
- else cmd in
+ match plugin_name with
+ | "vddk" | "curl" -> cmd
+ | _ -> Nbdkit.add_filter_if_available cmd "readahead" in
(* Caching extents speeds up qemu-img, especially its consecutive
* block_status requests with req_one=1.
--
2.25.0
10:13 a.m.
On 6/19/20 7:47 AM, Richard W.M. Jones wrote:
This appears to be the cause of timeouts during the conversion step
where VMware VCenter server's Tomcat HTTPS server stops responding to
requests (or rather, responds only with 503 errors). The server later
recovers and in fact because of the retry filter the conversion
usually succeeds, but I found that we can avoid the problem by
disabling readahead.
---
v2v/nbdkit_sources.ml | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
Is it just a matter of readahead requesting too much data at once? Is
it time to start revisiting the idea of moving the core of the
'blocksize' filter into the server itself, in order to make it easier
for various plugins and filters to request min/max block limits, and
where the common server code then stitches together read-modify-write or
splits requests as needed to the next layer?
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
10:36 a.m.
On Fri, Jun 19, 2020 at 10:13:05AM -0500, Eric Blake wrote:
On 6/19/20 7:47 AM, Richard W.M. Jones wrote:
>This appears to be the cause of timeouts during the conversion step
>where VMware VCenter server's Tomcat HTTPS server stops responding to
>requests (or rather, responds only with 503 errors). The server later
>recovers and in fact because of the retry filter the conversion
>usually succeeds, but I found that we can avoid the problem by
>disabling readahead.
>---
> v2v/nbdkit_sources.ml | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
Is it just a matter of readahead requesting too much data at once?
Is it time to start revisiting the idea of moving the core of the
'blocksize' filter into the server itself, in order to make it
easier for various plugins and filters to request min/max block
limits, and where the common server code then stitches together
read-modify-write or splits requests as needed to the next layer?
In fact this patch doesn't seem to have fixed the problem (it seemed
to fix it for me, but not for mxie).
However I do believe that our readahead filter is very naive and could
do with a rewrite, but blocksize in the server may not help. In
particular:
* virt-v2v <= 1.40 (when using qemu's curl driver) used to adjust the
qemu curl driver readahead setting between the conversion and
copying steps, essentially only applying readahead when copying.
Conversion has somewhat random access patterns, whereas copying is
mostly linear.
* We enable qemu's copyonread feature to cache reads in the overlay.
However this certainly interacts badly with nbdkit-readahead-filter
for a couple of reasons: (1) copyonread breaks linear reads, thus
defeating the filter heuristics. (2) the readahead filter will read
beyond the requested region, but it doesn't "know" if that data was
already cached in the overlay by a previous read, therefore it reads
more than necessary.
(It's also possible - but I didn't check - that these poor
interactions don't happen with qemu's curl driver readahead +
copyonread. At least, it's all running in the same binary.)
* VMware is a piece of crap and can't handle large reads without
crashing (neither over HTTPS nor VDDK as it happens). Here,
breaking up large reads could help, but we wouldn't want to bake
that into nbdkit curl plugin because it's only applicable when using
VMware's failing software, most HTTP servers are fine.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
1498
days inactive
1498
days old
3 comments
2 participants
participants (2)
-
Eric Blake -
Richard W.M. Jones