[Libguestfs] Performance regression in -o rhv-upload questions

Hi Nir, https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c4 I'm looking for some advice/help with a performance regression in virt-v2v between 1.44.2 and the latest version. It's very obvious and reproducible when I do a conversion from a local disk to local RHV using -o rhv-upload, specifically: $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc https://ovirt4410.home.annexia.org/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os ovirt-data -on test5 -of raw The guest is an ordinary Fedora guest containing some junk data. Now that I've been able to reproduce the problem locally, it turns out to be not at all what I thought it was going to be. The timings show that: - new virt-v2v is much faster doing the copy, but - new virt-v2v takes ages finalizing the transfer (about 10x longer than the old version) virt-v2v 1.44.2: Complete log: http://oirase.annexia.org/tmp/virt-v2v-1.44.2-rhv-upload.log [ 63.1] Copying disk 1/1 ... transfer 7aecd359-3706-49f5-8a0c-c8799f7b100a is finalizing_success transfer 7aecd359-3706-49f5-8a0c-c8799f7b100a is finished_success transfer 7aecd359-3706-49f5-8a0c-c8799f7b100a finalized in 6.105 seconds [ 89.2] Creating output metadata [ 89.8] Finishing off virt-v2v 1.45.97: Complete log: http://oirase.annexia.org/tmp/virt-v2v-1.45.97-rhv-upload.log [ 71.0] Copying disk 1/1 [ 82.7] Creating output metadata ... transfer 6ea3d724-16f9-4bda-a33e-69a783480abc is finalizing_success transfer 6ea3d724-16f9-4bda-a33e-69a783480abc is finished_success transfer 6ea3d724-16f9-4bda-a33e-69a783480abc finalized in 61.552 seconds [ 144.9] Finishing off It's not a completely like-for-like comparison because the rhv-upload backend changed a lot between these versions. In particular if I was going to pick some suspicious change it would be my refactoring here which was supposed to be neutral but maybe isn't: https://github.com/libguestfs/virt-v2v/commit/143a22860216b94d3a817061930... Unfortunately this commit is almost impossible to revert because of deep restructuring of surrounding code. Both branches have your earlier fix to finalization, so it shouldn't be this: https://github.com/libguestfs/virt-v2v/commit/79702b28329d15a7485801ed7e9... Another idea: Old virt-v2v uses qemu-img convert which does not flush by default. New virt-v2v uses nbdcopy with the --flush option, so it will call imageio PATCH /path ... "op":"flush" at the end. However removing nbdcopy --flush didn't help very much (only a couple of seconds off). Do you have any other ideas? What exactly does imageio do during the finalizing_success state? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW