[Libguestfs] Virt-v2v performance benchmarking part 1 [long]

This email is about looking at the historical performance of virt-v2v through the versions over the past year. The year has seen some very significant changes to virt-v2v meant to improve modularity, flexibility and in some cases performance. The main change that could affect performance is exposing the disk image pipelines of the input and output sides, and using nbdcopy (instead of qemu-img convert) to copy the data. So for the sake of this email when I talk about virt-v2v, I also mean nbdkit and libnbd which we've been co-developing. My first challenge is to try to make some kind of test case that we can use for testing conversion and copying performance. For this I'm using Linux. Because Windows conversions are always simpler than Linux conversions, and because Windows is proprietary, it's neither necessary or convenient to create a test case around Windows. I created a simple and reproducible pair of test cases by doing: $ mkdir /var/tmp/3G $ dd if=/dev/urandom of=/var/tmp/3G/1 bs=1M count=1024 $ cp /var/tmp/3G/1 /var/tmp/3G/2 $ cp /var/tmp/3G/1 /var/tmp/3G/3 $ virt-builder fedora-35 --copy-in /var/tmp/3G:/var/tmp $ virt-builder fedora-35 --copy-in /var/tmp/3G:/var/tmp --format=qcow2 $ ls -lsh fedora-35.* 3.9G -rw-r--r--. 1 rjones rjones 6.0G Jan 7 11:33 fedora-35.img 3.9G -rw-r--r--. 1 rjones rjones 3.9G Jan 7 11:37 fedora-35.qcow2 The test case should be locally reproducible for anyone wanting to try to reproduce the results. Also the amount of used disk space can be easily adjusted by using more or fewer 1GB random files, in case copying is too fast or slow on local hardware. The test case could then be copied to VMware or used locally, with virt-v2v commands like these: $ virt-v2v -i disk fedora-35.img -o null $ virt-v2v -i disk fedora-35.qcow2 -o null $ virt-v2v -i disk fedora-35.img -o local -os /var/tmp ; time sync $ virt-v2v -i disk fedora-35.qcow2 -o local -os /var/tmp ; time sync $ virt-v2v -ic esxi://... -it vddk [etc] fedora-35 [-o ...] A few remarks: - Old virt-v2v handled -o null very efficiently by throwing away the data inside qemu-img convert. Modular virt-v2v runs an nbdkit-null-plugin instance on the output side and always copies the data to it. This is always going to be a bit slower. Therefore I'm preferring the tests that write to local disk. - Old virt-v2v did not sync the data to disk, but nbdcopy always does, so with old virt-v2v you have to add a "time sync" command and include that in the copying time for fairness. - Raw and qcow2 use different code paths in the new virt-v2v, so benchmarking both is useful. - Copying from VMware can test raw over HTTP, VDDK, SSH which are all different code paths. I'm not benchmarking this at the moment. - Output to something other than null/local would also be interesting, but I'm not benchmarking that at the moment. interest for benchmarking performance: $ virt-v2v -i disk fedora-35.img -o null [ 1.1] Opening the source [ 7.3] Inspecting the source [ 12.4] Checking for sufficient free disk space in the guest [ 12.4] Converting Fedora Linux 35 (Thirty Five) to run on KVM [ 56.6] Mapping filesystem data to avoid copying unused and blank areas [ 59.8] Closing the overlay [ 60.3] Assigning disks to buses [ 60.3] Checking if the guest needs BIOS or UEFI to boot [ 61.3] Copying disk 1/1 █ 100% [****************************************] [ 62.4] Creating output metadata [ 62.4] Finishing off => Conversion time: 56.6 - 12.4 = 44.2 Mapping time: 59.8 - 56.6 = 3.2 Copying time: 62.4 - 61.3 = 1.1 (+ sync time for old virt-v2v) The "mapping time" is the time taken to run fstrim over the source. I include this because it regressed quite a lot just after we modularised virt-v2v and was later fixed, although it's still not as fast as old virt-v2v. Now what I am doing is running some tests over every virt-v2v we have released in CentOS 9 Stream over the last year. As mentioned above I also need to do libnbd & nbdkit since we were co-developing those, so I'm actually using RHEL 9 composes for this. However I am leaving libguestfs, libvirt and qemu at the newest versions throughout this because older versions had various problems running on current hardware which have since been fixed (my theory is that the versions of these should not make any significant difference). Also: $ virt-v2v --version virt-v2v: symbol lookup error: virt-v2v: undefined symbol: guestfs_list_9p This was easily fixed by adding a trivial LD_PRELOAD library which just loaded phony symbols for guestfs_list_9p & guestfs_mount_9p_argv (both dropped in later libguestfs). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v