virt-p2v VM testing bitrot?
I've started looking into RHBZ 1590721 for virt-p2v. For p2v development, quick local testing is helpful. "make run-virt-p2v-directly" seems to be working great; however, the VM-based test methods seem to have developed problems, since I last looked. Namely: - "make run-virt-p2v-in-a-vm" boots quickly, but the GUI does not come up. - "make run-virt-p2v-in-an-nvme-vm" boots *incredibly slowly*. I checked, and the host CPU utilization during guest boot was around 20% (and KVM was enabled). I don't understand how or why, but exposing the "physical machine" disk over NVMe slows guest boot to a crawl -- it looks strangely "IO-bound". I don't recall this from the time I added this Makefile target! Do these symptoms look familiar? (For RHBZ 1590721, I think run-virt-p2v-directly will suffice; just wanted to record the above somewhere.) Thanks Laszlo
On Wed, Aug 24, 2022 at 04:12:04PM +0200, Laszlo Ersek wrote:
I've started looking into RHBZ 1590721 for virt-p2v.
For p2v development, quick local testing is helpful. "make run-virt-p2v-directly" seems to be working great; however, the VM-based test methods seem to have developed problems, since I last looked.
Namely:
- "make run-virt-p2v-in-a-vm" boots quickly, but the GUI does not come up.
- "make run-virt-p2v-in-an-nvme-vm" boots *incredibly slowly*. I checked, and the host CPU utilization during guest boot was around 20% (and KVM was enabled). I don't understand how or why, but exposing the "physical machine" disk over NVMe slows guest boot to a crawl -- it looks strangely "IO-bound". I don't recall this from the time I added this Makefile target!
Do these symptoms look familiar?
Yes I had noticed this. I did look at it briefly but couldn't work out what was going wrong. Rich.
(For RHBZ 1590721, I think run-virt-p2v-directly will suffice; just wanted to record the above somewhere.)
Thanks Laszlo
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On 08/24/22 16:13, Richard W.M. Jones wrote:
On Wed, Aug 24, 2022 at 04:12:04PM +0200, Laszlo Ersek wrote:
I've started looking into RHBZ 1590721 for virt-p2v.
For p2v development, quick local testing is helpful. "make run-virt-p2v-directly" seems to be working great; however, the VM-based test methods seem to have developed problems, since I last looked.
Namely:
- "make run-virt-p2v-in-a-vm" boots quickly, but the GUI does not come up.
- "make run-virt-p2v-in-an-nvme-vm" boots *incredibly slowly*. I checked, and the host CPU utilization during guest boot was around 20% (and KVM was enabled). I don't understand how or why, but exposing the "physical machine" disk over NVMe slows guest boot to a crawl -- it looks strangely "IO-bound". I don't recall this from the time I added this Makefile target!
Do these symptoms look familiar?
Yes I had noticed this. I did look at it briefly but couldn't work out what was going wrong.
Thanks for confirming! I think I've found something, at least in relation to the first issue (i.e., run-virt-p2v-in-a-vm booting quickly, but not launching the p2v service). The problem seems to be the following: - Both the "physical machine" disk (fedora.img) and the bootable p2v disk (virt-p2v.img) are based on Fedora 35. The former because of commit 3b59d4acb82f ("Use fedora-35 as the test physical machine", 2022-05-10), the latter because my host system runs Fedora 35, and so virt-p2v-make-disk picks F35. - Consequently, both the "hd0" drive and the "usb0" drive are based on the same Fedora 35 virt-builder template. - The grub command line (part of *both* images) specifies the *same* kernel parameter "root=UUID=fae2...". - This filesystem UUID exists *twice* in the virtual machine however, once in "hd0" and another time in "usb0" -- duplicate UUID. - The guest kernel ends up mounting / loading the root filesystem from "hd0", that is, the "physical machine" Fedora 35 image, which does not have anything injected that's related to p2v. I found this by attempting to log in to the guest. The "p2v" password did not work (which is hardcoded in "virt-p2v-make-disk") , but the password printed by virt-builder under the PHYSICAL_MACHINE target in "Makefile" did work. Then, out of curiosity, I tried mounting the "other" disk in the running VM, but the XFS driver rejected the mount attempt, due to duplicate UUIDs. Now, virt-sysprep (and therefore virt-builder) has an operation called "fs-uuids", but it is not enabled by default: <https://libguestfs.org/virt-sysprep.1.html#notes-on-fs-uuids>, and we also don't enable it manually for either "fedora.img" or "virt-p2v.img". I guess the issue was "masked" before commit 3b59d4acb82f ("Use fedora-35 as the test physical machine", 2022-05-10), at least on my workstation. I think the Fedora versions I've used on this workstation have been F34 and F35, so for "virt-p2v.img", virt-builder would download F34 and F35 templates, which would differ from the template used for "fedora.img" (F32) -- hence the ultimately different fs UUIDs. ... I have no news regarding the NVMe slowness, but the above UUID collision exists in "run-virt-p2v-in-an-nvme-vm" just the same. Once we fix the UUID conflict, the NVMe test might go back to normal too. Laszlo
Rich.
(For RHBZ 1590721, I think run-virt-p2v-directly will suffice; just wanted to record the above somewhere.)
Thanks Laszlo
On Wed, Aug 24, 2022 at 05:25:50PM +0200, Laszlo Ersek wrote:
On 08/24/22 16:13, Richard W.M. Jones wrote:
On Wed, Aug 24, 2022 at 04:12:04PM +0200, Laszlo Ersek wrote:
I've started looking into RHBZ 1590721 for virt-p2v.
For p2v development, quick local testing is helpful. "make run-virt-p2v-directly" seems to be working great; however, the VM-based test methods seem to have developed problems, since I last looked.
Namely:
- "make run-virt-p2v-in-a-vm" boots quickly, but the GUI does not come up.
- "make run-virt-p2v-in-an-nvme-vm" boots *incredibly slowly*. I checked, and the host CPU utilization during guest boot was around 20% (and KVM was enabled). I don't understand how or why, but exposing the "physical machine" disk over NVMe slows guest boot to a crawl -- it looks strangely "IO-bound". I don't recall this from the time I added this Makefile target!
Do these symptoms look familiar?
Yes I had noticed this. I did look at it briefly but couldn't work out what was going wrong.
Thanks for confirming!
I think I've found something, at least in relation to the first issue (i.e., run-virt-p2v-in-a-vm booting quickly, but not launching the p2v service). The problem seems to be the following:
- Both the "physical machine" disk (fedora.img) and the bootable p2v disk (virt-p2v.img) are based on Fedora 35. The former because of commit 3b59d4acb82f ("Use fedora-35 as the test physical machine", 2022-05-10), the latter because my host system runs Fedora 35, and so virt-p2v-make-disk picks F35.
- Consequently, both the "hd0" drive and the "usb0" drive are based on the same Fedora 35 virt-builder template.
- The grub command line (part of *both* images) specifies the *same* kernel parameter "root=UUID=fae2...".
- This filesystem UUID exists *twice* in the virtual machine however, once in "hd0" and another time in "usb0" -- duplicate UUID.
- The guest kernel ends up mounting / loading the root filesystem from "hd0", that is, the "physical machine" Fedora 35 image, which does not have anything injected that's related to p2v.
Ah yes, I remember being extremely confused by why it was booting the wrong guest, and also I remember this exact bug having happened long ago in the past. Can we re-randomize the file system UUID of one of them when creating the test disks?
I found this by attempting to log in to the guest. The "p2v" password did not work (which is hardcoded in "virt-p2v-make-disk") , but the password printed by virt-builder under the PHYSICAL_MACHINE target in "Makefile" did work. Then, out of curiosity, I tried mounting the "other" disk in the running VM, but the XFS driver rejected the mount attempt, due to duplicate UUIDs.
Now, virt-sysprep (and therefore virt-builder) has an operation called "fs-uuids", but it is not enabled by default: <https://libguestfs.org/virt-sysprep.1.html#notes-on-fs-uuids>, and we also don't enable it manually for either "fedora.img" or "virt-p2v.img".
ISTR virt-sysprep fs-uuids doesn't really work because it won't update the grub config (or /etc/fstab). However it should be possible in general to do this with libguestfs.
I guess the issue was "masked" before commit 3b59d4acb82f ("Use fedora-35 as the test physical machine", 2022-05-10), at least on my workstation. I think the Fedora versions I've used on this workstation have been F34 and F35, so for "virt-p2v.img", virt-builder would download F34 and F35 templates, which would differ from the template used for "fedora.img" (F32) -- hence the ultimately different fs UUIDs.
... I have no news regarding the NVMe slowness, but the above UUID collision exists in "run-virt-p2v-in-an-nvme-vm" just the same. Once we fix the UUID conflict, the NVMe test might go back to normal too.
Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 08/24/22 17:55, Richard W.M. Jones wrote:
On Wed, Aug 24, 2022 at 05:25:50PM +0200, Laszlo Ersek wrote:
On 08/24/22 16:13, Richard W.M. Jones wrote:
On Wed, Aug 24, 2022 at 04:12:04PM +0200, Laszlo Ersek wrote:
I've started looking into RHBZ 1590721 for virt-p2v.
For p2v development, quick local testing is helpful. "make run-virt-p2v-directly" seems to be working great; however, the VM-based test methods seem to have developed problems, since I last looked.
Namely:
- "make run-virt-p2v-in-a-vm" boots quickly, but the GUI does not come up.
- "make run-virt-p2v-in-an-nvme-vm" boots *incredibly slowly*. I checked, and the host CPU utilization during guest boot was around 20% (and KVM was enabled). I don't understand how or why, but exposing the "physical machine" disk over NVMe slows guest boot to a crawl -- it looks strangely "IO-bound". I don't recall this from the time I added this Makefile target!
Do these symptoms look familiar?
Yes I had noticed this. I did look at it briefly but couldn't work out what was going wrong.
Thanks for confirming!
I think I've found something, at least in relation to the first issue (i.e., run-virt-p2v-in-a-vm booting quickly, but not launching the p2v service). The problem seems to be the following:
- Both the "physical machine" disk (fedora.img) and the bootable p2v disk (virt-p2v.img) are based on Fedora 35. The former because of commit 3b59d4acb82f ("Use fedora-35 as the test physical machine", 2022-05-10), the latter because my host system runs Fedora 35, and so virt-p2v-make-disk picks F35.
- Consequently, both the "hd0" drive and the "usb0" drive are based on the same Fedora 35 virt-builder template.
- The grub command line (part of *both* images) specifies the *same* kernel parameter "root=UUID=fae2...".
- This filesystem UUID exists *twice* in the virtual machine however, once in "hd0" and another time in "usb0" -- duplicate UUID.
- The guest kernel ends up mounting / loading the root filesystem from "hd0", that is, the "physical machine" Fedora 35 image, which does not have anything injected that's related to p2v.
Ah yes, I remember being extremely confused by why it was booting the wrong guest, and also I remember this exact bug having happened long ago in the past. Can we re-randomize the file system UUID of one of them when creating the test disks?
I'd like to implement it in a separate script. Do I have to list that script under EXTRA_DIST? What other Makefile macros? Thanks! Laszlo
On Thu, Aug 25, 2022 at 02:42:54PM +0200, Laszlo Ersek wrote:
I'd like to implement it in a separate script.
Do I have to list that script under EXTRA_DIST?
Usually yes, because we'd want it to go into the *.tar.gz file which is how some distros (eg. Fedora) consume virt-p2v. There is a useful rule: make dist && make maintainer-check-extra-dist which should check that all files necessary are listed in EXTRA_DIST (or pulled in implicitly - you don't need EXTRA_DIST for *_SOURCES and a few other things).
What other Makefile macros?
I'd do something like: check_SCRIPTS = my_script.sh which tells automake that 'my_script.sh' is needed for 'make check'. If the script needs @foo@ autoconf expansions, then create a file called 'my_script.sh.in' and add that to configure.ac. See existing AC_CONFIG_FILES entries. I think check_SCRIPTS and/or AC_CONFIG_FILES implicitly adds the script or its source to EXTRA_DIST. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On 08/25/22 15:06, Richard W.M. Jones wrote:
On Thu, Aug 25, 2022 at 02:42:54PM +0200, Laszlo Ersek wrote:
I'd like to implement it in a separate script.
Do I have to list that script under EXTRA_DIST?
Usually yes, because we'd want it to go into the *.tar.gz file which is how some distros (eg. Fedora) consume virt-p2v.
There is a useful rule:
make dist && make maintainer-check-extra-dist
which should check that all files necessary are listed in EXTRA_DIST (or pulled in implicitly - you don't need EXTRA_DIST for *_SOURCES and a few other things).
What other Makefile macros?
I'd do something like:
check_SCRIPTS = my_script.sh
which tells automake that 'my_script.sh' is needed for 'make check'.
If the script needs @foo@ autoconf expansions, then create a file called 'my_script.sh.in' and add that to configure.ac. See existing AC_CONFIG_FILES entries.
I think check_SCRIPTS and/or AC_CONFIG_FILES implicitly adds the script or its source to EXTRA_DIST.
Rich.
Thanks -- I added check_SCRIPTS, but then I also needed to extend EXTRA_DIST ("make maintainer-check-extra-dist" complained). Laszlo
participants (2)
-
Laszlo Ersek -
Richard W.M. Jones