[virt-v2v PATCH 0/4] Build and test suite fixes
After (1) applying the libnbd patch at <https://listman.redhat.com/archives/libguestfs/2021-September/msg00158.html>, (2) creating a symlink to the just-built nbdkit binary under ~/bin, (3) applying these virt-v2v patches, and (4) masking the in-place v2v test temporarily with SKIP_TEST_V2V_IN_PLACE_SH=1, I can run "make" and "make check" successfully in the virt-v2v tree. Thanks, Laszlo Laszlo Ersek (4): Makefile.am: use $(LIBNBD_LIBS) for linking OCaml programs test-v2v-cdrom: update the CD-ROM's bus to SATA in the converted domain test-v2v-i-ova: update the CD-ROM's bus to SATA in the converted domain test-v2v-i-ova: spell out viosock element in output domain XML convert/Makefile.am | 1 + output/Makefile.am | 1 + v2v/Makefile.am | 1 + tests/test-v2v-cdrom.expected | 2 +- tests/test-v2v-cdrom.xml.in | 4 +++- tests/test-v2v-i-ova.xml | 3 ++- 6 files changed, 9 insertions(+), 3 deletions(-) -- 2.19.1.3.g30247aa5d201
Otherwise the build fails with errors such as
GEN helper-v2v-output /usr/bin/ld: cannot find -lnbd collect2: error: ld returned 1 exit status File "caml_startup", line 1: Error: Error during linking (exit code 1)
in the "convert", "output" and "v2v" modules. (This patch is similar to guestfs-tools commit 9b0f2e2d5893 ("Makefile.am: use $(LIBGUESTFS_LIBS) for linking OCaml programs", 2021-09-06).) Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- convert/Makefile.am | 1 + output/Makefile.am | 1 + v2v/Makefile.am | 1 + 3 files changed, 3 insertions(+) diff --git a/convert/Makefile.am b/convert/Makefile.am index bb7c444f4aed..398bc1ed77a3 100644 --- a/convert/Makefile.am +++ b/convert/Makefile.am @@ -99,6 +99,7 @@ OCAMLCLIBS = \ $(LIBGUESTFS_LIBS) \ $(LIBXML2_LIBS) \ $(LIBOSINFO_LIBS) \ + $(LIBNBD_LIBS) \ -lgnu OCAMLFLAGS = $(OCAML_FLAGS) $(OCAML_WARN_ERROR) -ccopt '$(CFLAGS)' diff --git a/output/Makefile.am b/output/Makefile.am index 46e6560cb2bf..a63ad4c70d45 100644 --- a/output/Makefile.am +++ b/output/Makefile.am @@ -140,6 +140,7 @@ OCAMLCLIBS = \ $(LIBGUESTFS_LIBS) \ $(LIBXML2_LIBS) \ $(JANSSON_LIBS) \ + $(LIBNBD_LIBS) \ -lgnu OCAMLFLAGS = $(OCAML_FLAGS) $(OCAML_WARN_ERROR) -ccopt '$(CFLAGS)' diff --git a/v2v/Makefile.am b/v2v/Makefile.am index 1ce11814e193..f1073d96399f 100644 --- a/v2v/Makefile.am +++ b/v2v/Makefile.am @@ -83,6 +83,7 @@ OCAMLCLIBS = \ $(JANSSON_LIBS) \ $(LIBOSINFO_LIBS) \ $(LIBINTL) \ + $(LIBNBD_LIBS) \ -lgnu OCAMLFLAGS = $(OCAML_FLAGS) $(OCAML_WARN_ERROR) -ccopt '$(CFLAGS)' -- 2.19.1.3.g30247aa5d201
Per commit ac39fa292c31 ("v2v: Set machine type explicitly for outputs which support it (RHBZ#1581428).", 2020-12-04), Windows 7 guests (which are "more modern" than Windows XP guests) are converted to Q35, not I440FX boards. Per related commit d0267122e348 ("v2v: -o libvirt: Map IDE disks to bus="sata" on Q35.", 2020-12-04), when a Windows 7 guest with an IDE CD-ROM -- hence, an I440FX board -- is converted, the CD-ROM in the converted domain will be on the SATA bus (Q35 does not have an IDE controller, only a SATA controller). Because the Windows guest used in "test-v2v-cdrom" is "Windows 7 Phony Edition", it triggers the above cascade, and currently fails with:
--- ./test-v2v-cdrom.expected 2021-09-03 22:51:31.026185527 +0200 +++ test-v2v-cdrom.d/disks 2021-09-19 13:01:47.471745622 +0200 @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> ./test-v2v-cdrom.sh: unexpected disk assignments
The conversion seems correct, but the test expectation is stale. Most probably, commit d0267122e348 missed updating the test data. Do it now. Fixes: d0267122e348202f6fac4d833f7448409e7129c9 Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- tests/test-v2v-cdrom.expected | 2 +- tests/test-v2v-cdrom.xml.in | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/test-v2v-cdrom.expected b/tests/test-v2v-cdrom.expected index 34d2bf5961b0..17bd152d8e64 100644 --- a/tests/test-v2v-cdrom.expected +++ b/tests/test-v2v-cdrom.expected @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> diff --git a/tests/test-v2v-cdrom.xml.in b/tests/test-v2v-cdrom.xml.in index 6bad5eab1cd4..a6e1e3f514d5 100644 --- a/tests/test-v2v-cdrom.xml.in +++ b/tests/test-v2v-cdrom.xml.in @@ -35,7 +35,9 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='@abs_top_builddir@/test-data/phony-guests/blank-disk.img'/> - <!-- virt-v2v should preserve the device name and bus --> + <!-- virt-v2v should change the bus to sata, due to Windows 7 + triggering a machine type change from i440fx to q35. Beyond that, + virt-v2v should preserve the on-bus index. --> <target dev='hdc' bus='ide'/> </disk> </devices> -- 2.19.1.3.g30247aa5d201
On Mon, Sep 27, 2021 at 08:34:56AM +0200, Laszlo Ersek wrote:
Per commit ac39fa292c31 ("v2v: Set machine type explicitly for outputs which support it (RHBZ#1581428).", 2020-12-04), Windows 7 guests (which are "more modern" than Windows XP guests) are converted to Q35, not I440FX boards.
Per related commit d0267122e348 ("v2v: -o libvirt: Map IDE disks to bus="sata" on Q35.", 2020-12-04), when a Windows 7 guest with an IDE CD-ROM -- hence, an I440FX board -- is converted, the CD-ROM in the converted domain will be on the SATA bus (Q35 does not have an IDE controller, only a SATA controller).
Because the Windows guest used in "test-v2v-cdrom" is "Windows 7 Phony Edition", it triggers the above cascade, and currently fails with:
--- ./test-v2v-cdrom.expected 2021-09-03 22:51:31.026185527 +0200 +++ test-v2v-cdrom.d/disks 2021-09-19 13:01:47.471745622 +0200 @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> ./test-v2v-cdrom.sh: unexpected disk assignments
The conversion seems correct, but the test expectation is stale. Most probably, commit d0267122e348 missed updating the test data. Do it now.
Actually the conversion is wrong - we need to update configuration inside the guest. Which is why I didn't fix this test - to remind myself that there's still a bug to be fixed. https://bugzilla.redhat.com/show_bug.cgi?id=1637857 Rich.
Fixes: d0267122e348202f6fac4d833f7448409e7129c9 Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- tests/test-v2v-cdrom.expected | 2 +- tests/test-v2v-cdrom.xml.in | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/tests/test-v2v-cdrom.expected b/tests/test-v2v-cdrom.expected index 34d2bf5961b0..17bd152d8e64 100644 --- a/tests/test-v2v-cdrom.expected +++ b/tests/test-v2v-cdrom.expected @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> diff --git a/tests/test-v2v-cdrom.xml.in b/tests/test-v2v-cdrom.xml.in index 6bad5eab1cd4..a6e1e3f514d5 100644 --- a/tests/test-v2v-cdrom.xml.in +++ b/tests/test-v2v-cdrom.xml.in @@ -35,7 +35,9 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='@abs_top_builddir@/test-data/phony-guests/blank-disk.img'/> - <!-- virt-v2v should preserve the device name and bus --> + <!-- virt-v2v should change the bus to sata, due to Windows 7 + triggering a machine type change from i440fx to q35. Beyond that, + virt-v2v should preserve the on-bus index. --> <target dev='hdc' bus='ide'/> </disk> </devices> -- 2.19.1.3.g30247aa5d201
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On 09/27/21 10:30, Richard W.M. Jones wrote:
On Mon, Sep 27, 2021 at 08:34:56AM +0200, Laszlo Ersek wrote:
Per commit ac39fa292c31 ("v2v: Set machine type explicitly for outputs which support it (RHBZ#1581428).", 2020-12-04), Windows 7 guests (which are "more modern" than Windows XP guests) are converted to Q35, not I440FX boards.
Per related commit d0267122e348 ("v2v: -o libvirt: Map IDE disks to bus="sata" on Q35.", 2020-12-04), when a Windows 7 guest with an IDE CD-ROM -- hence, an I440FX board -- is converted, the CD-ROM in the converted domain will be on the SATA bus (Q35 does not have an IDE controller, only a SATA controller).
Because the Windows guest used in "test-v2v-cdrom" is "Windows 7 Phony Edition", it triggers the above cascade, and currently fails with:
--- ./test-v2v-cdrom.expected 2021-09-03 22:51:31.026185527 +0200 +++ test-v2v-cdrom.d/disks 2021-09-19 13:01:47.471745622 +0200 @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> ./test-v2v-cdrom.sh: unexpected disk assignments
The conversion seems correct, but the test expectation is stale. Most probably, commit d0267122e348 missed updating the test data. Do it now.
Actually the conversion is wrong - we need to update configuration inside the guest. Which is why I didn't fix this test - to remind myself that there's still a bug to be fixed.
Hmm, good point; now I remember that the i440fx->q35 machine type change is basically a motherboard replacement for Windows. So do we have to update some "windows device paths" (leading to the CD-ROM) in the registry? Thanks, Laszlo
https://bugzilla.redhat.com/show_bug.cgi?id=1637857
Rich.
Fixes: d0267122e348202f6fac4d833f7448409e7129c9 Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- tests/test-v2v-cdrom.expected | 2 +- tests/test-v2v-cdrom.xml.in | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/tests/test-v2v-cdrom.expected b/tests/test-v2v-cdrom.expected index 34d2bf5961b0..17bd152d8e64 100644 --- a/tests/test-v2v-cdrom.expected +++ b/tests/test-v2v-cdrom.expected @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> diff --git a/tests/test-v2v-cdrom.xml.in b/tests/test-v2v-cdrom.xml.in index 6bad5eab1cd4..a6e1e3f514d5 100644 --- a/tests/test-v2v-cdrom.xml.in +++ b/tests/test-v2v-cdrom.xml.in @@ -35,7 +35,9 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='@abs_top_builddir@/test-data/phony-guests/blank-disk.img'/> - <!-- virt-v2v should preserve the device name and bus --> + <!-- virt-v2v should change the bus to sata, due to Windows 7 + triggering a machine type change from i440fx to q35. Beyond that, + virt-v2v should preserve the on-bus index. --> <target dev='hdc' bus='ide'/> </disk> </devices> -- 2.19.1.3.g30247aa5d201
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
On Wed, Sep 29, 2021 at 09:07:57PM +0200, Laszlo Ersek wrote:
On 09/27/21 10:30, Richard W.M. Jones wrote:
On Mon, Sep 27, 2021 at 08:34:56AM +0200, Laszlo Ersek wrote:
Per commit ac39fa292c31 ("v2v: Set machine type explicitly for outputs which support it (RHBZ#1581428).", 2020-12-04), Windows 7 guests (which are "more modern" than Windows XP guests) are converted to Q35, not I440FX boards.
Per related commit d0267122e348 ("v2v: -o libvirt: Map IDE disks to bus="sata" on Q35.", 2020-12-04), when a Windows 7 guest with an IDE CD-ROM -- hence, an I440FX board -- is converted, the CD-ROM in the converted domain will be on the SATA bus (Q35 does not have an IDE controller, only a SATA controller).
Because the Windows guest used in "test-v2v-cdrom" is "Windows 7 Phony Edition", it triggers the above cascade, and currently fails with:
--- ./test-v2v-cdrom.expected 2021-09-03 22:51:31.026185527 +0200 +++ test-v2v-cdrom.d/disks 2021-09-19 13:01:47.471745622 +0200 @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> ./test-v2v-cdrom.sh: unexpected disk assignments
The conversion seems correct, but the test expectation is stale. Most probably, commit d0267122e348 missed updating the test data. Do it now.
Actually the conversion is wrong - we need to update configuration inside the guest. Which is why I didn't fix this test - to remind myself that there's still a bug to be fixed.
Hmm, good point; now I remember that the i440fx->q35 machine type change is basically a motherboard replacement for Windows.
So do we have to update some "windows device paths" (leading to the CD-ROM) in the registry?
We currently partially install the virtio block drivers in the Windows guest (just enough to get the guest to boot on the target), and Windows itself re-installs the virtio block driver and other drivers it needs, and that's enough to get it to see C: As for other hard disk partitions, Windows does indeed contain a mapping to other drives in the Registry but IIRC it's not sensitive to the device driver (unlike Linux /dev/vdX vs /dev/sdX). If you're interested in that, see libguestfs.git/daemon/inspect_fs_windows.ml: get_drive_mappings. We never bothered with attempting to handle conversion of floppy drives or CD-ROMs for Windows. On Linux we do better: We iterate over all the configuration files in /etc and change device paths. The significance of this bug is we need to change (eg) /dev/hdc to /dev/<something>. The difficulty is working out where the device will appear on the target and not having it conflict with any hard disk, something we partly control (see virt-v2v.git/convert/target_bus_assignment.ml*) Rich.
Thanks, Laszlo
https://bugzilla.redhat.com/show_bug.cgi?id=1637857
Rich.
Fixes: d0267122e348202f6fac4d833f7448409e7129c9 Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- tests/test-v2v-cdrom.expected | 2 +- tests/test-v2v-cdrom.xml.in | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/tests/test-v2v-cdrom.expected b/tests/test-v2v-cdrom.expected index 34d2bf5961b0..17bd152d8e64 100644 --- a/tests/test-v2v-cdrom.expected +++ b/tests/test-v2v-cdrom.expected @@ -4,5 +4,5 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hdc' bus='ide'/> + <target dev='sdc' bus='sata'/> </disk> diff --git a/tests/test-v2v-cdrom.xml.in b/tests/test-v2v-cdrom.xml.in index 6bad5eab1cd4..a6e1e3f514d5 100644 --- a/tests/test-v2v-cdrom.xml.in +++ b/tests/test-v2v-cdrom.xml.in @@ -35,7 +35,9 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='@abs_top_builddir@/test-data/phony-guests/blank-disk.img'/> - <!-- virt-v2v should preserve the device name and bus --> + <!-- virt-v2v should change the bus to sata, due to Windows 7 + triggering a machine type change from i440fx to q35. Beyond that, + virt-v2v should preserve the on-bus index. --> <target dev='hdc' bus='ide'/> </disk> </devices> -- 2.19.1.3.g30247aa5d201
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
(+libvirt-devel) On 09/29/21 21:22, Richard W.M. Jones wrote:
We currently partially install the virtio block drivers in the Windows guest (just enough to get the guest to boot on the target), and Windows itself re-installs the virtio block driver and other drivers it needs, and that's enough to get it to see C:
As for other hard disk partitions, Windows does indeed contain a mapping to other drives in the Registry but IIRC it's not sensitive to the device driver (unlike Linux /dev/vdX vs /dev/sdX). If you're interested in that, see libguestfs.git/daemon/inspect_fs_windows.ml: get_drive_mappings. We never bothered with attempting to handle conversion of floppy drives or CD-ROMs for Windows.
OK. So AIUI, that means no work is needed here for Windows.
On Linux we do better: We iterate over all the configuration files in /etc and change device paths. The significance of this bug is we need to change (eg) /dev/hdc to /dev/<something>. The difficulty is working out where the device will appear on the target and not having it conflict with any hard disk, something we partly control (see virt-v2v.git/convert/target_bus_assignment.ml*)
AIUI the conflict avoidance logic ("no overlapping disks") is already in place. The question is how to translate device paths in /etc/fstab and similar. Please correct me if I'm wrong: at the moment, I believe virt-v2v parses and manipulates the following elements and attributes in the domain XML: <target dev='hda' bus='ide'/> <target dev='hdb' bus='ide'/> <target dev='hdc' bus='ide'/> <target dev='hdd' bus='ide'/> ^^^ ^^^ My understanding is however that the target/@dev attribute is mostly irrelevant: https://libvirt.org/formatdomain.html#hard-drives-floppy-disks-cdroms The dev attribute indicates the "logical" device name. The actual device name specified is not guaranteed to map to the device name in the guest OS. Treat it as a device ordering hint. [...] What actually matters is the target/@bus attribute, in combination with the sibling element <address>. Such as: <target dev='hda' bus='ide'/> ^^^ <address type='drive' controller='0' bus='0' target='0' unit='0'/> ^ ^ ^ <target dev='hdb' bus='ide'/> ^^^ <address type='drive' controller='0' bus='0' target='0' unit='1'/> ^ ^ ^ <target dev='hdc' bus='ide'/> ^^^ <address type='drive' controller='0' bus='1' target='0' unit='0'/> ^ ^ ^ <target dev='hdd' bus='ide'/> ^^^ <address type='drive' controller='0' bus='1' target='0' unit='1'/> ^ ^ ^ So, target/@dev should be mostly ignored; what matters is the following tuple: (target/@bus, address/@controller, address/@bus, address/@unit) Extracting just the tuples: (ide, 0, 0, 0) (ide, 0, 0, 1) (ide, 0, 1, 0) (ide, 0, 1, 1) The first two components of each tuple -- i.e., (ide, 0) -- refer to the following IDE controller: <controller type='ide' index='0'> ^^^^^^^^^^ ^^^^^^^^^ <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> and then the rest of the components, such as (0, 0), (0, 1), (1, 0), (1, 1), identify the disk on that IDE controller. (Side comment: the PCI location of the (first) IDE controller is fixed in QEMU; if one tries to change it, libvirt complains: "Primary IDE controller must have PCI address 0:0:1.1".) (Side comment: on the QEMU command line, this maps to -device ide-cd,bus=ide.0,unit=0,... \ -device ide-cd,bus=ide.0,unit=1,... \ -device ide-cd,bus=ide.1,unit=0,... \ -device ide-cd,bus=ide.1,unit=1,... \ ) Inside the guest, /dev/hd* nodes don't even exist, so it's unlikely that /etc/fstab would refer to them. /etc/fstab can however refer to symlinks under "/dev/disk/by-id" (for example): lrwxrwxrwx. 1 root root 9 Sep 30 11:54 ata-QEMU_DVD-ROM_QM00001 -> ../../sr0 lrwxrwxrwx. 1 root root 9 Sep 30 11:54 ata-QEMU_DVD-ROM_QM00002 -> ../../sr1 lrwxrwxrwx. 1 root root 9 Sep 30 11:54 ata-QEMU_DVD-ROM_QM00003 -> ../../sr2 lrwxrwxrwx. 1 root root 9 Sep 30 11:54 ata-QEMU_DVD-ROM_QM00004 -> ../../sr3 Furthermore, we have pseudo-files (directories) such as: /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sr0 /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:1/0:0:1:0/block/sr1 /sys/devices/pci0000:00/0000:00:01.1/ata2/host1/target1:0:0/1:0:0:0/block/sr2 /sys/devices/pci0000:00/0000:00:01.1/ata2/host1/target1:0:1/1:0:1:0/block/sr3 ^ ^ So in order to map a device path from the original guest's "/etc/fstab", such as "/dev/disk/by-id/ata-QEMU_DVD-ROM_QM00003", to the original domain XML's <disk> element, we have to do the following in the "source" appliance: NODE=$(realpath /dev/disk/by-id/ata-QEMU_DVD-ROM_QM00003) # -> /dev/sr2 NODE=${NODE#/dev/} # -> sr2 DEVPATH=$(ls -d /sys/devices/pci0000:00/0000:00:01.1/ata?/host?/target?:0:?/?:0:?:0/block/$NODE) # -> /sys/devices/pci0000:00/0000:00:01.1/ata2/host1/target1:0:0/1:0:0:0/block/sr2 And then map the "1:0:0:0" pathname component from $DEVPATH to: <target dev='hdc' bus='ide'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> ^^^^^^^ ^^^^^^^^ [1]:0:0:0 1:0:[0]:0 in the original domain XML. This tells us under what device node the original guest sees the host-side file (<source> element). After conversion, on the Q35 board, the inverse mapping is needed. We start from the domain XML, <target dev='sd*' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> <target dev='sd*' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> <target dev='sd*' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> <target dev='sd*' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> <target dev='sd*' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='4'/> <target dev='sd*' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='5'/> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> (Side comment: the PCI B/D/F of the SATA controller is also fixed in QEMU; otherwise libvirt complains: "Primary SATA controller must have PCI address 0:0:1f.2".) (Side comment: the QEMU command line is -device ide-cd,bus=ide.0,... \ -device ide-cd,bus=ide.1,... \ -device ide-cd,bus=ide.2,... \ -device ide-cd,bus=ide.3,... \ -device ide-cd,bus=ide.4,... \ -device ide-cd,bus=ide.5,... \ ) In the guest we have: /sys/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sr0 /sys/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sr1 /sys/devices/pci0000:00/0000:00:1f.2/ata3/host2/target2:0:0/2:0:0:0/block/sr2 /sys/devices/pci0000:00/0000:00:1f.2/ata4/host3/target3:0:0/3:0:0:0/block/sr3 /sys/devices/pci0000:00/0000:00:1f.2/ata5/host4/target4:0:0/4:0:0:0/block/sr4 /sys/devices/pci0000:00/0000:00:1f.2/ata6/host5/target5:0:0/5:0:0:0/block/sr5 So, assuming we mapped the original (i440fx) "/dev/disk/by-id/ata-QEMU_DVD-ROM_QM00003" guest device path to some <source> element (= host-side file) in the original domain XML, and assuming virt-v2v assigned the same <source> element to the following element on the Q35 board: <target dev='sd*' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='4'/> ^^^^^^^^ we find the device node in the destination appliance as follows: NODE=$(basename /sys/devices/pci0000:00/0000:00:1f.2/ata?/host?/target?:0:0/4:0:0:0/block/*) ^ unit='4' # -> sr4 and then replace "/dev/disk/by-id/ata-QEMU_DVD-ROM_QM00003" with "/dev/sr4" in "/etc/fstab". All this requires virt-v2v to parse complete <address> elements from the original domain XML, and to generate complete <address> elements in the destination domain XML. Is that feasible? The @wwn and @serial attributes don't look safe to me, because the guest could refer to them even when the original domain XML does not spell them out (eg. "QM00003" is such a @serial). So neither can we trust that a @serial is present in the original XML, nor can we just go ahead and generate a @serial if it is absent. (The generation step could immediately break references such as "QM00003" in the guest.) /dev/disk/by-label and /dev/disk/by-uuid are based on media contents, and multiple CD-ROMs may (read-only) map the same host-side file, so those are not good for mapping either, I think. Also does not cover CD-ROM devices that are empty (have no medium) at the time of conversion, but "/etc/fstab" still refers to them (potentially with "noauto"). So I think the only reliable ID is the hardware device path. ... Now if *that* needs to work when the original guest comes from a different management application than libvirt, then I have no idea. The original address (PCI B/D/F of the IDE controller, and IDE bus and unit of the drive) need to be known somehow; otherwise we cannot associate the guest-side "/dev/..." reference with the host-side file underlying that CD-ROM. Thanks, Laszlo
On Thu, Sep 30, 2021 at 01:12:39PM +0200, Laszlo Ersek wrote:
All this requires virt-v2v to parse complete <address> elements from the original domain XML, and to generate complete <address> elements in the destination domain XML. Is that feasible?
The input is not always (in fact, hardly ever) full libvirt XML. It's input specific to the hypervisor. For VMware it might be: - the *.vmx file (the real source of truth) (-i vmx) - partial libvirt XML generated by libvirt's vpx driver, but this is derived from information from VMware APIs and ultimately that comes from the *.vmx file (-i libvirt -ic esx:// or -ic vpx://) - the *.ovf file (-i ova) - nothing at all! (-i disk) Also we don't currently try to find or rewrite /dev/disk/ paths in guest configuration files. The only rewriting that happens is for /dev/[hs]d* block device filenames and a few others. The actual code that does this is convert/convert_linux.ml:remap_block_devices So I wouldn't over-think this. It's likely fine to identify such devices and rewrite them as "/dev/cdrom", assuming that (I didn't check) udev creates that symlink for any reasonably modern Linux. And if there's more than one attached CD to the source, only convert the first one and warn about but drop the others. Note that the aim of virt-v2v is to make it boot on the target, make sure the network works, and get the user to a login prompt. The MTV management app around virt-v2v allows site-specific pre- and post- configuration to happen (site-specific Ansible playbooks). There also exists ssh and remote console. For people using virt-v2v on the command line, if they get the guest to boot on the target and not every device fully works, there is an expectation that they can log in as root and fix small[*] things. Rich. [*] Obviously if the boot / system is completely broken, not that. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Thu, Sep 30, 2021 at 12:53:51PM +0100, Richard W.M. Jones wrote:
On Thu, Sep 30, 2021 at 01:12:39PM +0200, Laszlo Ersek wrote:
All this requires virt-v2v to parse complete <address> elements from the original domain XML, and to generate complete <address> elements in the destination domain XML. Is that feasible?
Just to put a bit of flesh on these bones, links to the source:
The input is not always (in fact, hardly ever) full libvirt XML. It's input specific to the hypervisor. For VMware it might be:
- the *.vmx file (the real source of truth) (-i vmx)
Here's the virt-v2v code that can parse either a local or remote (over ssh) *.vmx file into virt-v2v's internal metadata representation: https://github.com/libguestfs/virt-v2v/blob/master/input/parse_domain_from_v... Libvirt also has a vmx driver, but I decided not to use it because it just adds an extra translation step and probably loses fidelity in the process. In all cases, the *.vmx file is the source of truth for VMware.
- partial libvirt XML generated by libvirt's vpx driver, but this is derived from information from VMware APIs and ultimately that comes from the *.vmx file (-i libvirt -ic esx:// or -ic vpx://)
The libvirt driver: https://gitlab.com/libvirt/libvirt/-/tree/master/src/esx I've hacked on this one a little bit in the past, and ... it's complicated.
- the *.ovf file (-i ova)
OVF is a pseudo-standard invented by VMware, but essentially a plot so they can say they are "standard" when in fact it's just a data dump of internal VMware structures with no compatibility across software that generates or consumes OVF. Here's our OVF parser (for VMware-OVF): https://github.com/libguestfs/virt-v2v/blob/master/input/OVF.ml We can also generate OVF, but we generate the RHV-flavour of OVF which (see above) is not related except that some XML element names overlap.
- nothing at all! (-i disk)
If you give virt-v2v just a disk then it will invent some metadata: https://github.com/libguestfs/virt-v2v/blob/5adf437bcc00586961d8aa058559f86e... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Hi Rich, (dropping libvirt-devel) On 09/30/21 13:53, Richard W.M. Jones wrote:
Also we don't currently try to find or rewrite /dev/disk/ paths in guest configuration files. The only rewriting that happens is for /dev/[hs]d* block device filenames and a few others. The actual code that does this is convert/convert_linux.ml:remap_block_devices
So I wouldn't over-think this. It's likely fine to identify such devices and rewrite them as "/dev/cdrom", assuming that (I didn't check) udev creates that symlink for any reasonably modern Linux. And if there's more than one attached CD to the source, only convert the first one and warn about but drop the others.
So I've started reading "convert/convert_linux.ml" in parallel with the OCaml manual. (I dabbled for a few weeks in Haskell when everyone else did a few years ago, so it's not 100% unfamiliar.) Question: let family = match inspect.i_distro with | "fedora" | "rhel" | "centos" | "scientificlinux" | "redhat-based" | "oraclelinux" -> `RHEL_family | "altlinux" -> `ALT_family | "sles" | "suse-based" | "opensuse" -> `SUSE_family | "debian" | "ubuntu" | "linuxmint" | "kalilinux" -> `Debian_family Here, RHEL_family, ALT_family, SUSE_family, Debian_family are not plain constructors ("fixed" variants) [1] but polymorphic ones [2]. Why? Was this done *only* in order so that an explicit type definition such as type os_family = RHEL_family | ALT_family | SUSE_family | Debian_family could be avoided? Based on my (incomplete understanding) of the ocaml docs, this looks like a bad idea. We need no polymorphism here, and using a static type (a fixed variant) is generally beneficial [3]. The following (minimally modified) definition at the OCaml REPL: let family1 distro = match distro with | "fedora" | "rhel" | "centos" | "scientificlinux" | "redhat-based" | "oraclelinux" -> `RHEL_family | "altlinux" -> `ALT_family | "sles" | "suse-based" | "opensuse" -> `SUSE_family | "debian" | "ubuntu" | "linuxmint" | "kalilinux" -> `Debian_family | _ -> assert false;; deduces the following type: string -> [> `ALT_family | `Debian_family | `RHEL_family | `SUSE_family ] = <fun> with the nasty "[>" mark at the start (allowing for further type refinement [2], which I think we don't need here). Conversely. type os_family = RHEL_family | ALT_family | SUSE_family | Debian_family;; let family2 distro = match distro with | "fedora" | "rhel" | "centos" | "scientificlinux" | "redhat-based" | "oraclelinux" -> RHEL_family | "altlinux" -> ALT_family | "sles" | "suse-based" | "opensuse" -> SUSE_family | "debian" | "ubuntu" | "linuxmint" | "kalilinux" -> Debian_family | _ -> assert false;; (note the explicit os_family type definition, and the removal of the backticks from the constructor names) comes back with the type: val family2 : string -> os_family = <fun> Basically converting a string to an enum constant. So, what's the reason for the polymorphic variant? [1] https://ocaml.org/manual/coreexamples.html#s%3Atut-recvariants [2] https://ocaml.org/manual/polyvariant.html#sec48 [3] https://ocaml.org/manual/polyvariant.html#s%3Apolyvariant-weaknesses Thanks! Laszlo
On Fri, Oct 01, 2021 at 12:35:50PM +0200, Laszlo Ersek wrote:
Hi Rich,
(dropping libvirt-devel)
On 09/30/21 13:53, Richard W.M. Jones wrote:
Also we don't currently try to find or rewrite /dev/disk/ paths in guest configuration files. The only rewriting that happens is for /dev/[hs]d* block device filenames and a few others. The actual code that does this is convert/convert_linux.ml:remap_block_devices
So I wouldn't over-think this. It's likely fine to identify such devices and rewrite them as "/dev/cdrom", assuming that (I didn't check) udev creates that symlink for any reasonably modern Linux. And if there's more than one attached CD to the source, only convert the first one and warn about but drop the others.
So I've started reading "convert/convert_linux.ml" in parallel with the OCaml manual. (I dabbled for a few weeks in Haskell when everyone else did a few years ago, so it's not 100% unfamiliar.)
Question:
let family = match inspect.i_distro with | "fedora" | "rhel" | "centos" | "scientificlinux" | "redhat-based" | "oraclelinux" -> `RHEL_family | "altlinux" -> `ALT_family | "sles" | "suse-based" | "opensuse" -> `SUSE_family | "debian" | "ubuntu" | "linuxmint" | "kalilinux" -> `Debian_family
Here, RHEL_family, ALT_family, SUSE_family, Debian_family are not plain constructors ("fixed" variants) [1] but polymorphic ones [2].
Why?
Was this done *only* in order so that an explicit type definition such as
type os_family = RHEL_family | ALT_family | SUSE_family | Debian_family
could be avoided?
There are two types of variants in OCaml. The first is the normal one and it requires a type definition. eg: type t = A | B | C of int | D of string * int with values like “A” or “C 3” or “D ("hello", 1)”. These are ones we'd normally use and have the (arguable?) advantage that you have to write (and therefore, document) the full type somewhere. The second type are polymorphic variants. These are actually way more complicated than I'm going to describe below because they have a load of set theory behind them (if you really want to know see the docs that you've already linked to). They are written with backtick before the name of the variant. The brief idea is that you let the OCaml compiler work out the type for you. For example: let f = function 1 -> `A | 2 -> `B | i -> `C i creates a function f which returns the type: type pv = [ `A | `B | `C of int ] Actually if you type this into the toplevel it says: $ rlwrap ocaml OCaml version 4.12.0 # let f = function 1 -> `A | 2 -> `B | i -> `C i ;; val f : int -> [> `A | `B | `C of int ] = <fun> As you observe below there's an extra ">" character. That means that the type is "at least" that one, and there might be more cases, but the function that I wrote only contains the 3 cases. Anyway the short of this is that these open-ended, automatically resolved types don't require a type definition so they're less documented but more flexible. (And another thing: It's possible to reuse the constructors in unrelated types. eg. `A might be part of another type. But don't do this unless it's for a very good reason). In all cases the OCaml compiler enforces type safety, so although these types are less explicit, they're still safe. If I extend my example above and forget about one of the cases, the compiler will warn (in the v2v code, we've set up the flags to give a hard error): # let g j = match f j with `A -> 1 | `B -> 2 ;; Warning 8 [partial-match]: this pattern-matching is not exhaustive. Here is an example of a case that is not matched: `C _ val g : int -> int = <fun>
Based on my (incomplete understanding) of the ocaml docs, this looks like a bad idea. We need no polymorphism here, and using a static type (a fixed variant) is generally beneficial [3].
The following (minimally modified) definition at the OCaml REPL:
let family1 distro = match distro with | "fedora" | "rhel" | "centos" | "scientificlinux" | "redhat-based" | "oraclelinux" -> `RHEL_family | "altlinux" -> `ALT_family | "sles" | "suse-based" | "opensuse" -> `SUSE_family | "debian" | "ubuntu" | "linuxmint" | "kalilinux" -> `Debian_family | _ -> assert false;;
deduces the following type:
string -> [> `ALT_family | `Debian_family | `RHEL_family | `SUSE_family ] = <fun>
with the nasty "[>" mark at the start (allowing for further type refinement [2], which I think we don't need here). Conversely.
type os_family = RHEL_family | ALT_family | SUSE_family | Debian_family;;
let family2 distro = match distro with | "fedora" | "rhel" | "centos" | "scientificlinux" | "redhat-based" | "oraclelinux" -> RHEL_family | "altlinux" -> ALT_family | "sles" | "suse-based" | "opensuse" -> SUSE_family | "debian" | "ubuntu" | "linuxmint" | "kalilinux" -> Debian_family | _ -> assert false;;
(note the explicit os_family type definition, and the removal of the backticks from the constructor names) comes back with the type:
val family2 : string -> os_family = <fun>
Yes, this is all fair comment.
Basically converting a string to an enum constant.
So, what's the reason for the polymorphic variant?
I guess you could look at git history, but most likely it was "just because". It's a bit quicker and adding a new cases involves less code churn. I'm not opposed if you want to make the change above if you think it improves readability. In general I try to avoid tricksy features of OCaml, like functors, and this could (arguably) be one of those cases. BTW as an aside the implementation of the two types of constructors is completely different, and polymorphic variants have a slower implementation. For ordinary constructors they generally map to integers, which makes them very efficient (eg. they can be stored in registers etc). Polymorphic variants use a hash of the name instead making them slower overall. (see table in https://rwmj.wordpress.com/2009/08/05/ocaml-internals-part-2-strings-and-oth...) But a fun thing is you can cause hash collisions (although it is picked up by the compiler - it's still type safe!): # type collision = [`Eric_Cooper | `azdwbie];; Error: Variant tags `azdwbie and `Eric_Cooper have the same hash value. Change one of them. (https://groups.google.com/g/fa.caml/c/68oTntkRLyU) There's even a link-time pass to find hash collisions across the whole program. Rich.
[1] https://ocaml.org/manual/coreexamples.html#s%3Atut-recvariants [2] https://ocaml.org/manual/polyvariant.html#sec48 [3] https://ocaml.org/manual/polyvariant.html#s%3Apolyvariant-weaknesses
Thanks! Laszlo
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Fri, Oct 01, 2021 at 12:03:22PM +0100, Richard W.M. Jones wrote:
BTW as an aside the implementation of the two types of constructors is completely different, and polymorphic variants have a slower implementation. For ordinary constructors they generally map to integers, which makes them very efficient (eg. they can be stored in registers etc). Polymorphic variants use a hash of the name instead making them slower overall.
I mean it's slower not because of the hash - which is computed at compile time - but because various tricks cannot be used at runtime which are applicable to small ints. eg. For the types: type t = AAA | BBB | CCC type pv = [`AAA | `BBB | `CCC] and functions: let ft = function AAA -> 0 | BBB -> 1 | CCC -> 2 let fpv = function `AAA -> 0 | `BBB -> 1 | `CCC -> 2 the first function is an identity (because the input and output have the same runtime representation). It only forces the LSB to be set, for unclear reasons, I think this may be a missed optimization: camlTest__ft_86: sarq $1, %rax ; %rax /= 2 leaq 1(%rax,%rax), %rax ; %rax = 2*%rax+1 ret but the second function requires a bunch of work to deal with the hash values: camlTest__fpv_89: cmpq $6593797, %rax je .L101 cmpq $6693703, %rax jl .L102 movl $5, %eax ; return OCaml int 5 = C int 2 ret .align 4 .L102: movl $1, %eax ; return OCaml int 1 = C int 0 ret .align 4 .L101: movl $3, %eax ; return OCaml int 3 = C int 1 ret Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Fri, Oct 01, 2021 at 12:21:29PM +0100, Richard W.M. Jones wrote:
movl $5, %eax ; return OCaml int 5 = C int 2
I mean C int 5 = OCaml int 2, etc. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 10/01/21 13:03, Richard W.M. Jones wrote:
On Fri, Oct 01, 2021 at 12:35:50PM +0200, Laszlo Ersek wrote:
Hi Rich,
(dropping libvirt-devel)
On 09/30/21 13:53, Richard W.M. Jones wrote:
Also we don't currently try to find or rewrite /dev/disk/ paths in guest configuration files. The only rewriting that happens is for /dev/[hs]d* block device filenames and a few others. The actual code that does this is convert/convert_linux.ml:remap_block_devices
So I wouldn't over-think this. It's likely fine to identify such devices and rewrite them as "/dev/cdrom", assuming that (I didn't check) udev creates that symlink for any reasonably modern Linux. And if there's more than one attached CD to the source, only convert the first one and warn about but drop the others.
So I've started reading "convert/convert_linux.ml" in parallel with the OCaml manual. (I dabbled for a few weeks in Haskell when everyone else did a few years ago, so it's not 100% unfamiliar.)
Question:
let family = match inspect.i_distro with | "fedora" | "rhel" | "centos" | "scientificlinux" | "redhat-based" | "oraclelinux" -> `RHEL_family | "altlinux" -> `ALT_family | "sles" | "suse-based" | "opensuse" -> `SUSE_family | "debian" | "ubuntu" | "linuxmint" | "kalilinux" -> `Debian_family
Here, RHEL_family, ALT_family, SUSE_family, Debian_family are not plain constructors ("fixed" variants) [1] but polymorphic ones [2].
Why?
Was this done *only* in order so that an explicit type definition such as
type os_family = RHEL_family | ALT_family | SUSE_family | Debian_family
could be avoided?
There are two types of variants in OCaml. The first is the normal one and it requires a type definition. eg:
type t = A | B | C of int | D of string * int
with values like “A” or “C 3” or “D ("hello", 1)”.
These are ones we'd normally use and have the (arguable?) advantage that you have to write (and therefore, document) the full type somewhere.
The second type are polymorphic variants. These are actually way more complicated than I'm going to describe below because they have a load of set theory behind them (if you really want to know see the docs that you've already linked to). They are written with backtick before the name of the variant. The brief idea is that you let the OCaml compiler work out the type for you.
For example:
let f = function 1 -> `A | 2 -> `B | i -> `C i
creates a function f which returns the type:
type pv = [ `A | `B | `C of int ]
Actually if you type this into the toplevel it says:
$ rlwrap ocaml OCaml version 4.12.0
# let f = function 1 -> `A | 2 -> `B | i -> `C i ;; val f : int -> [> `A | `B | `C of int ] = <fun>
As you observe below there's an extra ">" character. That means that the type is "at least" that one, and there might be more cases, but the function that I wrote only contains the 3 cases.
Anyway the short of this is that these open-ended, automatically resolved types don't require a type definition so they're less documented but more flexible.
(And another thing: It's possible to reuse the constructors in unrelated types. eg. `A might be part of another type. But don't do this unless it's for a very good reason).
In all cases the OCaml compiler enforces type safety, so although these types are less explicit, they're still safe. If I extend my example above and forget about one of the cases, the compiler will warn (in the v2v code, we've set up the flags to give a hard error):
# let g j = match f j with `A -> 1 | `B -> 2 ;; Warning 8 [partial-match]: this pattern-matching is not exhaustive. Here is an example of a case that is not matched: `C _ val g : int -> int = <fun>
Based on my (incomplete understanding) of the ocaml docs, this looks like a bad idea. We need no polymorphism here, and using a static type (a fixed variant) is generally beneficial [3].
The following (minimally modified) definition at the OCaml REPL:
let family1 distro = match distro with | "fedora" | "rhel" | "centos" | "scientificlinux" | "redhat-based" | "oraclelinux" -> `RHEL_family | "altlinux" -> `ALT_family | "sles" | "suse-based" | "opensuse" -> `SUSE_family | "debian" | "ubuntu" | "linuxmint" | "kalilinux" -> `Debian_family | _ -> assert false;;
deduces the following type:
string -> [> `ALT_family | `Debian_family | `RHEL_family | `SUSE_family ] = <fun>
with the nasty "[>" mark at the start (allowing for further type refinement [2], which I think we don't need here). Conversely.
type os_family = RHEL_family | ALT_family | SUSE_family | Debian_family;;
let family2 distro = match distro with | "fedora" | "rhel" | "centos" | "scientificlinux" | "redhat-based" | "oraclelinux" -> RHEL_family | "altlinux" -> ALT_family | "sles" | "suse-based" | "opensuse" -> SUSE_family | "debian" | "ubuntu" | "linuxmint" | "kalilinux" -> Debian_family | _ -> assert false;;
(note the explicit os_family type definition, and the removal of the backticks from the constructor names) comes back with the type:
val family2 : string -> os_family = <fun>
Yes, this is all fair comment.
Basically converting a string to an enum constant.
So, what's the reason for the polymorphic variant?
I guess you could look at git history, but most likely it was "just because". It's a bit quicker and adding a new cases involves less code churn.
I'm not opposed if you want to make the change above if you think it improves readability.
Thanks for the help! I don't want to leave a trail of such patches marking my learning of OCaml :) Just wanted to understand the reason. Laszlo
In general I try to avoid tricksy features of OCaml, like functors, and this could (arguably) be one of those cases.
BTW as an aside the implementation of the two types of constructors is completely different, and polymorphic variants have a slower implementation. For ordinary constructors they generally map to integers, which makes them very efficient (eg. they can be stored in registers etc). Polymorphic variants use a hash of the name instead making them slower overall.
(see table in https://rwmj.wordpress.com/2009/08/05/ocaml-internals-part-2-strings-and-oth...)
But a fun thing is you can cause hash collisions (although it is picked up by the compiler - it's still type safe!):
# type collision = [`Eric_Cooper | `azdwbie];; Error: Variant tags `azdwbie and `Eric_Cooper have the same hash value. Change one of them.
(https://groups.google.com/g/fa.caml/c/68oTntkRLyU)
There's even a link-time pass to find hash collisions across the whole program.
Rich.
[1] https://ocaml.org/manual/coreexamples.html#s%3Atut-recvariants [2] https://ocaml.org/manual/polyvariant.html#sec48 [3] https://ocaml.org/manual/polyvariant.html#s%3Apolyvariant-weaknesses
Thanks! Laszlo
On Thu, Sep 30, 2021 at 01:12:39PM +0200, Laszlo Ersek wrote:
On 09/29/21 21:22, Richard W.M. Jones wrote: Please correct me if I'm wrong: at the moment, I believe virt-v2v parses and manipulates the following elements and attributes in the domain XML:
<target dev='hda' bus='ide'/> <target dev='hdb' bus='ide'/> <target dev='hdc' bus='ide'/> <target dev='hdd' bus='ide'/> ^^^ ^^^
My understanding is however that the target/@dev attribute is mostly irrelevant:
https://libvirt.org/formatdomain.html#hard-drives-floppy-disks-cdroms
The dev attribute indicates the "logical" device name. The actual device name specified is not guaranteed to map to the device name in the guest OS. Treat it as a device ordering hint. [...]
I won't say it is irrelevant. Functionally @dev is absolutely still important, as it influences how the disk is attached to the VM. Rather I would say that the @dev attribute is misleading to users, because they mistakenly think it provides a guarantee that the disk will appear with this name inside the guest.
What actually matters is the target/@bus attribute, in combination with the sibling element <address>. Such as:
<target dev='hda' bus='ide'/> ^^^ <address type='drive' controller='0' bus='0' target='0' unit='0'/> ^ ^ ^
<target dev='hdb' bus='ide'/> ^^^ <address type='drive' controller='0' bus='0' target='0' unit='1'/> ^ ^ ^
<target dev='hdc' bus='ide'/> ^^^ <address type='drive' controller='0' bus='1' target='0' unit='0'/> ^ ^ ^
<target dev='hdd' bus='ide'/> ^^^ <address type='drive' controller='0' bus='1' target='0' unit='1'/> ^ ^ ^
So, target/@dev should be mostly ignored; what matters is the following tuple:
(target/@bus, address/@controller, address/@bus, address/@unit)
Yes, the <address/> is what libvirt internally drivers all configuration off, but in practice application developers almost never use the <address> element directly. They will just give target/@dev and libvirt will use that to automatically populate an <address/> element, in order to reliably fixate the guest ABI thereafter.
Extracting just the tuples:
(ide, 0, 0, 0) (ide, 0, 0, 1) (ide, 0, 1, 0) (ide, 0, 1, 1)
The first two components of each tuple -- i.e., (ide, 0) -- refer to the following IDE controller:
<controller type='ide' index='0'> ^^^^^^^^^^ ^^^^^^^^^ <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller>
and then the rest of the components, such as (0, 0), (0, 1), (1, 0), (1, 1), identify the disk on that IDE controller.
Yes, that's correct for IDE.
(Side comment: the PCI location of the (first) IDE controller is fixed in QEMU; if one tries to change it, libvirt complains: "Primary IDE controller must have PCI address 0:0:1.1".)
Yep, its defined by the QEMU machine type and we just have to accept that for IDE/SATA.
Inside the guest, /dev/hd* nodes don't even exist, so it's unlikely that /etc/fstab would refer to them. /etc/fstab can however refer to symlinks under "/dev/disk/by-id" (for example):
lrwxrwxrwx. 1 root root 9 Sep 30 11:54 ata-QEMU_DVD-ROM_QM00001 -> ../../sr0 lrwxrwxrwx. 1 root root 9 Sep 30 11:54 ata-QEMU_DVD-ROM_QM00002 -> ../../sr1 lrwxrwxrwx. 1 root root 9 Sep 30 11:54 ata-QEMU_DVD-ROM_QM00003 -> ../../sr2 lrwxrwxrwx. 1 root root 9 Sep 30 11:54 ata-QEMU_DVD-ROM_QM00004 -> ../../sr3
Furthermore, we have pseudo-files (directories) such as:
/sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:0/0:0:0:0/block/sr0 /sys/devices/pci0000:00/0000:00:01.1/ata1/host0/target0:0:1/0:0:1:0/block/sr1 /sys/devices/pci0000:00/0000:00:01.1/ata2/host1/target1:0:0/1:0:0:0/block/sr2 /sys/devices/pci0000:00/0000:00:01.1/ata2/host1/target1:0:1/1:0:1:0/block/sr3 ^ ^
So in order to map a device path from the original guest's "/etc/fstab", such as "/dev/disk/by-id/ata-QEMU_DVD-ROM_QM00003", to the original domain XML's <disk> element, we have to do the following in the "source" appliance:
NODE=$(realpath /dev/disk/by-id/ata-QEMU_DVD-ROM_QM00003) # -> /dev/sr2 NODE=${NODE#/dev/} # -> sr2 DEVPATH=$(ls -d /sys/devices/pci0000:00/0000:00:01.1/ata?/host?/target?:0:?/?:0:?:0/block/$NODE) # -> /sys/devices/pci0000:00/0000:00:01.1/ata2/host1/target1:0:0/1:0:0:0/block/sr2
And then map the "1:0:0:0" pathname component from $DEVPATH to:
<target dev='hdc' bus='ide'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> ^^^^^^^ ^^^^^^^^ [1]:0:0:0 1:0:[0]:0
in the original domain XML. This tells us under what device node the original guest sees the host-side file (<source> element).
Yes, to map from the guest to the libvirt XML, you need to be working in terms of the hardware buses/addresses.
So, assuming we mapped the original (i440fx) "/dev/disk/by-id/ata-QEMU_DVD-ROM_QM00003" guest device path to some <source> element (= host-side file) in the original domain XML, and assuming virt-v2v assigned the same <source> element to the following element on the Q35 board:
<target dev='sd*' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='4'/> ^^^^^^^^
we find the device node in the destination appliance as follows:
NODE=$(basename /sys/devices/pci0000:00/0000:00:1f.2/ata?/host?/target?:0:0/4:0:0:0/block/*) ^ unit='4' # -> sr4
and then replace "/dev/disk/by-id/ata-QEMU_DVD-ROM_QM00003" with "/dev/sr4" in "/etc/fstab".
I wouldn't recommend using any of the /dev/* devices, as those are all unstable when faced with changed guest configuration over time. /etc/fstab should really use a stable /dev/disk/ symlink, so it can be directly associated with the desired device based on the hardware topology, rather than guest OS device probe order.
/dev/disk/by-label and /dev/disk/by-uuid are based on media contents, and multiple CD-ROMs may (read-only) map the same host-side file, so those are not good for mapping either, I think. Also does not cover CD-ROM devices that are empty (have no medium) at the time of conversion, but "/etc/fstab" still refers to them (potentially with "noauto"). So I think the only reliable ID is the hardware device path.
Yep, the symlink based on hardware topology is the only thing that can be stable and unique. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Using the argument from the "test-v2v-cdrom: update the CD-ROM's bus to SATA in the converted domain" patch earlier in this series. Fixes: d0267122e348202f6fac4d833f7448409e7129c9 Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- tests/test-v2v-i-ova.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test-v2v-i-ova.xml b/tests/test-v2v-i-ova.xml index 671476b95d96..a018fd44aee8 100644 --- a/tests/test-v2v-i-ova.xml +++ b/tests/test-v2v-i-ova.xml @@ -28,7 +28,7 @@ </disk> <disk device='cdrom' type='file'> <driver name='qemu' type='raw'/> - <target dev='hda' bus='ide'/> + <target dev='sda' bus='sata'/> </disk> <disk device='floppy' type='file'> <driver name='qemu' type='raw'/> -- 2.19.1.3.g30247aa5d201
This is likely an omission from commit 05f780c16f01 ("v2v: support configuration of viosock driver", 2021-02-26) -- if the converted Windows guest does not dispose over the virtio-sock guest drivers, an explicit "none"-model element is now generated in the domain XML. Fixes: 05f780c16f0135c657615520c2245b42de1efc3e Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- tests/test-v2v-i-ova.xml | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test-v2v-i-ova.xml b/tests/test-v2v-i-ova.xml index a018fd44aee8..489c0ab09bcc 100644 --- a/tests/test-v2v-i-ova.xml +++ b/tests/test-v2v-i-ova.xml @@ -46,6 +46,7 @@ <backend model='random'>/dev/urandom</backend> </rng> <memballoon model='virtio'/> + <viosock model='none'/> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <console type='pty'/> -- 2.19.1.3.g30247aa5d201
On Mon, Sep 27, 2021 at 08:34:58AM +0200, Laszlo Ersek wrote:
This is likely an omission from commit 05f780c16f01 ("v2v: support configuration of viosock driver", 2021-02-26) -- if the converted Windows guest does not dispose over the virtio-sock guest drivers, an explicit "none"-model element is now generated in the domain XML.
Fixes: 05f780c16f0135c657615520c2245b42de1efc3e Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- tests/test-v2v-i-ova.xml | 1 + 1 file changed, 1 insertion(+)
diff --git a/tests/test-v2v-i-ova.xml b/tests/test-v2v-i-ova.xml index a018fd44aee8..489c0ab09bcc 100644 --- a/tests/test-v2v-i-ova.xml +++ b/tests/test-v2v-i-ova.xml @@ -46,6 +46,7 @@ <backend model='random'>/dev/urandom</backend> </rng> <memballoon model='virtio'/> + <viosock model='none'/> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <console type='pty'/>
Patches 1 & 4 are good. Patches 2 & 3 are symptoms of the incomplete fix: https://bugzilla.redhat.com/show_bug.cgi?id=1581428 https://bugzilla.redhat.com/show_bug.cgi?id=1637857 so need some different work (although fixing the test will be part of the solution). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 09/27/21 10:32, Richard W.M. Jones wrote:
On Mon, Sep 27, 2021 at 08:34:58AM +0200, Laszlo Ersek wrote:
This is likely an omission from commit 05f780c16f01 ("v2v: support configuration of viosock driver", 2021-02-26) -- if the converted Windows guest does not dispose over the virtio-sock guest drivers, an explicit "none"-model element is now generated in the domain XML.
Fixes: 05f780c16f0135c657615520c2245b42de1efc3e Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- tests/test-v2v-i-ova.xml | 1 + 1 file changed, 1 insertion(+)
diff --git a/tests/test-v2v-i-ova.xml b/tests/test-v2v-i-ova.xml index a018fd44aee8..489c0ab09bcc 100644 --- a/tests/test-v2v-i-ova.xml +++ b/tests/test-v2v-i-ova.xml @@ -46,6 +46,7 @@ <backend model='random'>/dev/urandom</backend> </rng> <memballoon model='virtio'/> + <viosock model='none'/> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <console type='pty'/>
Patches 1 & 4 are good.
Thanks! I pushed them as commits 3767ebca30c5 and 5adf437bcc00 respectively. I also synthesized an Acked-by in your name for each commit, from the above "good". (If I shouldn't do that, please let me know.) Thanks, Laszlo
Patches 2 & 3 are symptoms of the incomplete fix:
https://bugzilla.redhat.com/show_bug.cgi?id=1581428 https://bugzilla.redhat.com/show_bug.cgi?id=1637857
so need some different work (although fixing the test will be part of the solution).
Rich.
participants (3)
-
Daniel P. Berrangé -
Laszlo Ersek -
Richard W.M. Jones