Fwd: virt-v2v creating image that does not install guest agent on first boot
This message got caught in moderation because the attachment was large. I put the attachment here: http://oirase.annexia.org/tmp/2023-lgarrett-virt-v2v-debug-output.txt ----- Forwarded message from Lee Garrett <lgarrett@rocketjump.eu> ----- Date: Sun, 10 Sep 2023 00:09:17 +0200 From: Lee Garrett <lgarrett@rocketjump.eu> To: libguestfs@redhat.com Subject: virt-v2v creating image that does not install guest agent on first boot Hi, rwmjones from #guestfs on libera IRC pointed me to this mail address. I've noticed that converting the latest Windows 11 trial image via virt-v2v does not install the guest agent any more on first boot. I have let the machine settle for ~10 minutes to ensure it's not just because I'm too impatient. I can see C:\qemu-ga-x86-64.msi, and also a bunch of files in "C:\Program Files\Guestfs\Firstboot\" and .\Scripts, so the files are definitely copied onto the VM image, just not executed for some reason. The vmware image was downloaded from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/. The issue should be 100% reproducible (at least for me). I recall that an older image from two, three weeks ago downloaded from the same location did not have this issue. There guest agent was installed and working on the first boot. Unfortunately I overwrote the image so I cannot verify this. I have attached the debug output of the virt-v2v run of the latest image where this issue is apparent (guest agent not being installed). Please let me know if there's anything else I can check, or if you need any further info. Cheers, Lee ----- End forwarded message ----- -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Sun, Sep 10, 2023 at 11:34:22AM +0100, Richard W.M. Jones wrote:
This message got caught in moderation because the attachment was large. I put the attachment here:
http://oirase.annexia.org/tmp/2023-lgarrett-virt-v2v-debug-output.txt
----- Forwarded message from Lee Garrett <lgarrett@rocketjump.eu> -----
Date: Sun, 10 Sep 2023 00:09:17 +0200 From: Lee Garrett <lgarrett@rocketjump.eu> To: libguestfs@redhat.com Subject: virt-v2v creating image that does not install guest agent on first boot
Hi,
rwmjones from #guestfs on libera IRC pointed me to this mail address.
I've noticed that converting the latest Windows 11 trial image via virt-v2v does not install the guest agent any more on first boot. I have let the machine settle for ~10 minutes to ensure it's not just because I'm too impatient. I can see C:\qemu-ga-x86-64.msi, and also a bunch of files in "C:\Program Files\Guestfs\Firstboot\" and .\Scripts, so the files are definitely copied onto the VM image, just not executed for some reason.
I can see from the log that these are installed in the image.
The vmware image was downloaded from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/. The issue should be 100% reproducible (at least for me). I recall that an older image from two, three weeks ago downloaded from the same location did not have this issue. There guest agent was installed and working on the first boot. Unfortunately I overwrote the image so I cannot verify this.
I have attached the debug output of the virt-v2v run of the latest image where this issue is apparent (guest agent not being installed).
Please let me know if there's anything else I can check, or if you need any further info.
The main thing to look for is any debugging output which should appear in c:\program files\guestfs\firstboot\log.txt It would be interesting to know if this file is present at all, and what exactly it contains. I've CC'd my colleague Ming on this to see if she has noticed anything with recent Windows 11. Rich.
Cheers, Lee
----- End forwarded message -----
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Hi, On 11.09.23 15:13, Richard W.M. Jones wrote:
On Sun, Sep 10, 2023 at 11:34:22AM +0100, Richard W.M. Jones wrote:
This message got caught in moderation because the attachment was large. I put the attachment here:
http://oirase.annexia.org/tmp/2023-lgarrett-virt-v2v-debug-output.txt
----- Forwarded message from Lee Garrett <lgarrett@rocketjump.eu> -----
Date: Sun, 10 Sep 2023 00:09:17 +0200 From: Lee Garrett <lgarrett@rocketjump.eu> To: libguestfs@redhat.com Subject: virt-v2v creating image that does not install guest agent on first boot
Hi,
rwmjones from #guestfs on libera IRC pointed me to this mail address.
I've noticed that converting the latest Windows 11 trial image via virt-v2v does not install the guest agent any more on first boot. I have let the machine settle for ~10 minutes to ensure it's not just because I'm too impatient. I can see C:\qemu-ga-x86-64.msi, and also a bunch of files in "C:\Program Files\Guestfs\Firstboot\" and .\Scripts, so the files are definitely copied onto the VM image, just not executed for some reason.
I can see from the log that these are installed in the image.
The vmware image was downloaded from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/. The issue should be 100% reproducible (at least for me). I recall that an older image from two, three weeks ago downloaded from the same location did not have this issue. There guest agent was installed and working on the first boot. Unfortunately I overwrote the image so I cannot verify this.
I have attached the debug output of the virt-v2v run of the latest image where this issue is apparent (guest agent not being installed).
Please let me know if there's anything else I can check, or if you need any further info.
The main thing to look for is any debugging output which should appear in c:\program files\guestfs\firstboot\log.txt
It would be interesting to know if this file is present at all, and what exactly it contains.
There is unfortunately no log.txt in that or related paths.
I've CC'd my colleague Ming on this to see if she has noticed anything with recent Windows 11.
Rich.
Cheers, Lee
----- End forwarded message -----
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
If we suspect that the whole firstboot mechanism might not be working with the new version of Windows, one way to test it (on this one, or a freshly installed Windows VM) would be: $ virt-customize -a windows.img --firstboot-command 'echo hello' and see if "hello" is written in some form to the log.txt file inside the guest after it boots. If that doesn't work then it's likely some change in Windows which is breaking firstboot support. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
After testing it again today, the guest agent is still not installed, however this time the C:\Program Files\Guestfs\Firstboot\log.txt exists: starting firstboot service running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0001-wait-pnp.bat" 1 file(s) moved. Wait for PnP to complete .... exit code 0 running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat" 1 file(s) moved. Removing any previously scheduled qemu-ga installation ERROR: The system cannot find the file specified. Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi SUCCESS: The scheduled task "Firstboot-qemu-ga" has successfully been created. .... exit code 0 running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0003-uninstall-VMware-Tools.bat" 1 file(s) moved. uninstalling VMware Tools .... exit code 0 uninstalling firstboot service Service uninstalled successfully The folder scripts is empty, all scripts are in scripts-done. The contents of the folder are: 5000-0001-wait-pnp.bat 123 5000-0001-wait-pnp.log 5000-0002-install-qemu-ga-x86_64-msi-ps1.bat 5000-0003-uninstall-VMware-Tools.bat 5000-0003-uninstall-VMware-Tools.log Note that the .log is missing for qemu-ga. At least today I could twice provision the windows 11 VM, and in both cases the main log.txt was generated by the time I checked. How are the services scheduled? Could it be that they were delayed by > 1h? On 11.09.23 21:29, Richard W.M. Jones wrote:
If we suspect that the whole firstboot mechanism might not be working with the new version of Windows, one way to test it (on this one, or a freshly installed Windows VM) would be:
$ virt-customize -a windows.img --firstboot-command 'echo hello'
and see if "hello" is written in some form to the log.txt file inside the guest after it boots.
If that doesn't work then it's likely some change in Windows which is breaking firstboot support.
Rich.
Regards, Lee
On Thu, Sep 21, 2023 at 04:50:26PM +0200, Lee Garrett wrote:
After testing it again today, the guest agent is still not installed, however this time the C:\Program Files\Guestfs\Firstboot\log.txt exists:
starting firstboot service running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0001-wait-pnp.bat" 1 file(s) moved. Wait for PnP to complete .... exit code 0 running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat" 1 file(s) moved. Removing any previously scheduled qemu-ga installation ERROR: The system cannot find the file specified. Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi SUCCESS: The scheduled task "Firstboot-qemu-ga" has successfully been created. .... exit code 0 running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0003-uninstall-VMware-Tools.bat" 1 file(s) moved. uninstalling VMware Tools .... exit code 0 uninstalling firstboot service Service uninstalled successfully
The folder scripts is empty, all scripts are in scripts-done. The contents of the folder are: 5000-0001-wait-pnp.bat 123 5000-0001-wait-pnp.log 5000-0002-install-qemu-ga-x86_64-msi-ps1.bat 5000-0003-uninstall-VMware-Tools.bat 5000-0003-uninstall-VMware-Tools.log
Note that the .log is missing for qemu-ga.
This is what we'd expect, except maybe the missing log for qemu-ga. It's a little odd that it is missing, but it's also possible that the log file doesn't get written if there's no output.
At least today I could twice provision the windows 11 VM, and in both cases the main log.txt was generated by the time I checked. How are the services scheduled? Could it be that they were delayed by > 1h?
Yes, we do actually try to delay qemu-ga installation. It should only be delayed for about 120 seconds (not 1 hour). https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7... However there have been timezone bugs with this in the past (and maybe there still are), see the sordid history in these two links: https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7... Which TZ are you in? Rich.
On 11.09.23 21:29, Richard W.M. Jones wrote:
If we suspect that the whole firstboot mechanism might not be working with the new version of Windows, one way to test it (on this one, or a freshly installed Windows VM) would be:
$ virt-customize -a windows.img --firstboot-command 'echo hello'
and see if "hello" is written in some form to the log.txt file inside the guest after it boots.
If that doesn't work then it's likely some change in Windows which is breaking firstboot support.
Rich.
Regards, Lee
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 21.09.23 18:08, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 04:50:26PM +0200, Lee Garrett wrote:
After testing it again today, the guest agent is still not installed, however this time the C:\Program Files\Guestfs\Firstboot\log.txt exists:
starting firstboot service running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0001-wait-pnp.bat" 1 file(s) moved. Wait for PnP to complete .... exit code 0 running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat" 1 file(s) moved. Removing any previously scheduled qemu-ga installation ERROR: The system cannot find the file specified. Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi SUCCESS: The scheduled task "Firstboot-qemu-ga" has successfully been created. .... exit code 0 running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0003-uninstall-VMware-Tools.bat" 1 file(s) moved. uninstalling VMware Tools .... exit code 0 uninstalling firstboot service Service uninstalled successfully
The folder scripts is empty, all scripts are in scripts-done. The contents of the folder are: 5000-0001-wait-pnp.bat 123 5000-0001-wait-pnp.log 5000-0002-install-qemu-ga-x86_64-msi-ps1.bat 5000-0003-uninstall-VMware-Tools.bat 5000-0003-uninstall-VMware-Tools.log
Note that the .log is missing for qemu-ga.
This is what we'd expect, except maybe the missing log for qemu-ga. It's a little odd that it is missing, but it's also possible that the log file doesn't get written if there's no output.
At least today I could twice provision the windows 11 VM, and in both cases the main log.txt was generated by the time I checked. How are the services scheduled? Could it be that they were delayed by > 1h?
Yes, we do actually try to delay qemu-ga installation. It should only be delayed for about 120 seconds (not 1 hour).
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
However there have been timezone bugs with this in the past (and maybe there still are), see the sordid history in these two links:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
I see. That's not reassuring. :(
Which TZ are you in?
The host machine is UTC+2 (Berlin summer time), however the Windows 11 trial vmware image itself defaults to UTC-8 (Pacific Time US & Canada). After a little bit of digging this indeed seems to be the issue. `schtasks.exe /Query` gives: PS C:\Users\User> schtasks.exe /Query Folder: \ TaskName Next Run Time Status ======================================== ====================== =============== Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready [...] which is ~7 hours in the future. Btw, running `.\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat` by hand works just fine. So it definitely is just the scheduling issue that's preventing it from running.
Rich.
On 11.09.23 21:29, Richard W.M. Jones wrote:
If we suspect that the whole firstboot mechanism might not be working with the new version of Windows, one way to test it (on this one, or a freshly installed Windows VM) would be:
$ virt-customize -a windows.img --firstboot-command 'echo hello'
and see if "hello" is written in some form to the log.txt file inside the guest after it boots.
If that doesn't work then it's likely some change in Windows which is breaking firstboot support.
Rich.
Regards, Lee
Regards, Lee
On Thu, Sep 21, 2023 at 06:24:40PM +0200, Lee Garrett wrote:
On 21.09.23 18:08, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 04:50:26PM +0200, Lee Garrett wrote:
After testing it again today, the guest agent is still not installed, however this time the C:\Program Files\Guestfs\Firstboot\log.txt exists:
starting firstboot service running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0001-wait-pnp.bat" 1 file(s) moved. Wait for PnP to complete .... exit code 0 running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat" 1 file(s) moved. Removing any previously scheduled qemu-ga installation ERROR: The system cannot find the file specified. Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi SUCCESS: The scheduled task "Firstboot-qemu-ga" has successfully been created. .... exit code 0 running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0003-uninstall-VMware-Tools.bat" 1 file(s) moved. uninstalling VMware Tools .... exit code 0 uninstalling firstboot service Service uninstalled successfully
The folder scripts is empty, all scripts are in scripts-done. The contents of the folder are: 5000-0001-wait-pnp.bat 123 5000-0001-wait-pnp.log 5000-0002-install-qemu-ga-x86_64-msi-ps1.bat 5000-0003-uninstall-VMware-Tools.bat 5000-0003-uninstall-VMware-Tools.log
Note that the .log is missing for qemu-ga.
This is what we'd expect, except maybe the missing log for qemu-ga. It's a little odd that it is missing, but it's also possible that the log file doesn't get written if there's no output.
At least today I could twice provision the windows 11 VM, and in both cases the main log.txt was generated by the time I checked. How are the services scheduled? Could it be that they were delayed by > 1h?
Yes, we do actually try to delay qemu-ga installation. It should only be delayed for about 120 seconds (not 1 hour).
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
However there have been timezone bugs with this in the past (and maybe there still are), see the sordid history in these two links:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
I see. That's not reassuring. :(
Which TZ are you in?
The host machine is UTC+2 (Berlin summer time), however the Windows 11 trial vmware image itself defaults to UTC-8 (Pacific Time US & Canada).
After a little bit of digging this indeed seems to be the issue.
`schtasks.exe /Query` gives:
PS C:\Users\User> schtasks.exe /Query
Folder: \ TaskName Next Run Time Status ======================================== ====================== =============== Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready [...]
which is ~7 hours in the future. Btw, running `.\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat` by hand works just fine. So it definitely is just the scheduling issue that's preventing it from running.
So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area. Rich.
Rich.
On 11.09.23 21:29, Richard W.M. Jones wrote:
If we suspect that the whole firstboot mechanism might not be working with the new version of Windows, one way to test it (on this one, or a freshly installed Windows VM) would be:
$ virt-customize -a windows.img --firstboot-command 'echo hello'
and see if "hello" is written in some form to the log.txt file inside the guest after it boots.
If that doesn't work then it's likely some change in Windows which is breaking firstboot support.
Rich.
Regards, Lee
Regards, Lee
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On 21.09.23 19:43, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 06:24:40PM +0200, Lee Garrett wrote:
On 21.09.23 18:08, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 04:50:26PM +0200, Lee Garrett wrote:
After testing it again today, the guest agent is still not installed, however this time the C:\Program Files\Guestfs\Firstboot\log.txt exists:
starting firstboot service running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0001-wait-pnp.bat" 1 file(s) moved. Wait for PnP to complete .... exit code 0 running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat" 1 file(s) moved. Removing any previously scheduled qemu-ga installation ERROR: The system cannot find the file specified. Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi SUCCESS: The scheduled task "Firstboot-qemu-ga" has successfully been created. .... exit code 0 running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0003-uninstall-VMware-Tools.bat" 1 file(s) moved. uninstalling VMware Tools .... exit code 0 uninstalling firstboot service Service uninstalled successfully
The folder scripts is empty, all scripts are in scripts-done. The contents of the folder are: 5000-0001-wait-pnp.bat 123 5000-0001-wait-pnp.log 5000-0002-install-qemu-ga-x86_64-msi-ps1.bat 5000-0003-uninstall-VMware-Tools.bat 5000-0003-uninstall-VMware-Tools.log
Note that the .log is missing for qemu-ga.
This is what we'd expect, except maybe the missing log for qemu-ga. It's a little odd that it is missing, but it's also possible that the log file doesn't get written if there's no output.
At least today I could twice provision the windows 11 VM, and in both cases the main log.txt was generated by the time I checked. How are the services scheduled? Could it be that they were delayed by > 1h?
Yes, we do actually try to delay qemu-ga installation. It should only be delayed for about 120 seconds (not 1 hour).
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
However there have been timezone bugs with this in the past (and maybe there still are), see the sordid history in these two links:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
I see. That's not reassuring. :(
Which TZ are you in?
The host machine is UTC+2 (Berlin summer time), however the Windows 11 trial vmware image itself defaults to UTC-8 (Pacific Time US & Canada).
After a little bit of digging this indeed seems to be the issue.
`schtasks.exe /Query` gives:
PS C:\Users\User> schtasks.exe /Query
Folder: \ TaskName Next Run Time Status ======================================== ====================== =============== Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready [...]
which is ~7 hours in the future. Btw, running `.\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat` by hand works just fine. So it definitely is just the scheduling issue that's preventing it from running.
So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area.
It's 2.2.0-1 from Debian (12) bookworm. I've verified that it doesn't have any distro-specific patches. (https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian would have a patches/series file in this case)
Rich.
Rich.
On 11.09.23 21:29, Richard W.M. Jones wrote:
If we suspect that the whole firstboot mechanism might not be working with the new version of Windows, one way to test it (on this one, or a freshly installed Windows VM) would be:
$ virt-customize -a windows.img --firstboot-command 'echo hello'
and see if "hello" is written in some form to the log.txt file inside the guest after it boots.
If that doesn't work then it's likely some change in Windows which is breaking firstboot support.
Rich.
Regards, Lee
Regards, Lee
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote:
On 21.09.23 19:43, Richard W.M. Jones wrote:
So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area.
It's 2.2.0-1 from Debian (12) bookworm. I've verified that it doesn't have any distro-specific patches.
(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian would have a patches/series file in this case)
The timezone fixes are: commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100 convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000 convert/convert_windows.ml: Handle date formats with dots instead of / They are all included in >= 2.0 I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later. Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way? The code we run is here: https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7... Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote:
On 21.09.23 19:43, Richard W.M. Jones wrote:
So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area.
It's 2.2.0-1 from Debian (12) bookworm. I've verified that it doesn't have any distro-specific patches.
(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian would have a patches/series file in this case)
The timezone fixes are:
commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100
convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell
commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000
convert/convert_windows.ml: Handle date formats with dots instead of /
They are all included in >= 2.0
I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later.
Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way?
Or in even simpler terms, what is the time (and timezone) that this ^^^ machine was booted? Rich.
The code we run is here:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On 22.09.23 14:54, Richard W.M. Jones wrote:
On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote:
On 21.09.23 19:43, Richard W.M. Jones wrote:
So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area.
It's 2.2.0-1 from Debian (12) bookworm. I've verified that it doesn't have any distro-specific patches.
(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian would have a patches/series file in this case)
The timezone fixes are:
commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100
convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell
commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000
convert/convert_windows.ml: Handle date formats with dots instead of /
They are all included in >= 2.0
I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later.
Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way?
Or in even simpler terms, what is the time (and timezone) that this ^^^ machine was booted?
I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future. So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*. From what I can see, the host machine time zone is irrelevant (UTC+2). I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?
Rich.
The code we run is here:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 22.09.23 16:47, Lee Garrett wrote:
On 22.09.23 14:54, Richard W.M. Jones wrote:
On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote:
On 21.09.23 19:43, Richard W.M. Jones wrote:
So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area.
It's 2.2.0-1 from Debian (12) bookworm. I've verified that it doesn't have any distro-specific patches.
(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian would have a patches/series file in this case)
The timezone fixes are:
commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100
convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell
commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000
convert/convert_windows.ml: Handle date formats with dots instead of /
They are all included in >= 2.0
I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later.
Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way?
Or in even simpler terms, what is the time (and timezone) that this ^^^ machine was booted?
I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future.
So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*.
From what I can see, the host machine time zone is irrelevant (UTC+2).
I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?
I digged a bit into the XML description and noticed this: <clock offset="utc"/> To my knowledge Windows runs the BIOS clock in local time. So This should probably be <clock offset="localtime"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock> as this is what libvirt uses in the preset for Windows 11 hosts. I manually changed this before first boot, however it didn't solve the issue. It only increased the offset to +9 hours in the future (7 hours difference to UTC, +2 hours of local timezone). How does the firstboot injection exactly happen? Is the guest actually booted for it?
Rich.
The code we run is here:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Fri, Sep 22, 2023 at 05:27:25PM +0200, Lee Garrett wrote:
On 22.09.23 16:47, Lee Garrett wrote:
On 22.09.23 14:54, Richard W.M. Jones wrote:
On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote:
On 21.09.23 19:43, Richard W.M. Jones wrote:
So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area.
It's 2.2.0-1 from Debian (12) bookworm. I've verified that it doesn't have any distro-specific patches.
(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian would have a patches/series file in this case)
The timezone fixes are:
commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100
convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell
commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000
convert/convert_windows.ml: Handle date formats with dots instead of /
They are all included in >= 2.0
I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later.
Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way?
Or in even simpler terms, what is the time (and timezone) that this ^^^ machine was booted?
I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future.
So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*.
From what I can see, the host machine time zone is irrelevant (UTC+2).
I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?
I digged a bit into the XML description and noticed this:
<clock offset="utc"/>
To my knowledge Windows runs the BIOS clock in local time. So This should probably be
<clock offset="localtime"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock>
as this is what libvirt uses in the preset for Windows 11 hosts. I manually changed this before first boot, however it didn't solve the issue. It only increased the offset to +9 hours in the future (7 hours difference to UTC, +2 hours of local timezone). How does the firstboot injection exactly happen? Is the guest actually booted for it?
No, the guest is not running. Rich.
Rich.
The code we run is here:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On 9/22/23 16:47, Lee Garrett wrote:
On 22.09.23 14:54, Richard W.M. Jones wrote:
On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote:
On 21.09.23 19:43, Richard W.M. Jones wrote:
So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area.
It's 2.2.0-1 from Debian (12) bookworm. I've verified that it doesn't have any distro-specific patches.
(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian would have a patches/series file in this case)
The timezone fixes are:
commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100
convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell
commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000
convert/convert_windows.ml: Handle date formats with dots instead of /
They are all included in >= 2.0
I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later.
Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way?
Or in even simpler terms, what is the time (and timezone) that this ^^^ machine was booted?
I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future.
So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*.
From what I can see, the host machine time zone is irrelevant (UTC+2).
I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?
Right, I think there is a timezone disagreement between how we format the timestamp and how schtasks.exe takes it. What matters here is the /ST (start time) flag. Today we have (in the common submodule): add "$d = (get-date).AddSeconds(120)"; add "$dtfinfo = [System.Globalization.DateTimeFormatInfo]::CurrentInfo"; add "$sdp = $dtfinfo.ShortDatePattern"; add "$sdp = $sdp -replace 'y+', 'yyyy'"; add "$sdp = $sdp -replace 'M+', 'MM'"; add "$sdp = $sdp -replace 'd+', 'dd'"; add "schtasks.exe /Create /SC ONCE `"; add " /ST $d.ToString('HH:mm') /SD $d.ToString($sdp) `"; add " /RU SYSTEM /TN Firstboot-qemu-ga `"; add (sprintf " /TR \"C:\\%s /forcerestart /qn /l+*vx C:\\%s.log\"" msi_path msi_path); Note that for the /ST option's argument, we only perform the following steps: $d = (get-date).AddSeconds(120) /ST $d.ToString('HH:mm') This actually goes back to commit dc66e78fa37d ("windows: delay installation of qemu-ga MSI", 2020-03-10). The timestamp massaging we've since done only targeted the /SD (start date) option, not the start time (/ST) one! So the problem may be that (get-date).AddSeconds(120).ToString('HH:mm') formats the hour:minute timestamp in UTC (why though?), but the /ST option takes hour:minute in local time. Interestingly, DateTime objects seem to have a "Kind" property, which may be Utc, Local, or Unspec. https://learn.microsoft.com/en-us/dotnet/api/system.datetime.kind It seems to be used when converting from UTC to local or vice versa, and it probably influences how ToString() behaves too. I thought "get-date" returned a Local one, and /ST took a local one as well, but perhaps "get-date" returns a UTC timestamp in this case (when run from the script)? Laszlo
Rich.
The code we run is here:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 23.09.23 19:37, Laszlo Ersek wrote:
On 9/22/23 16:47, Lee Garrett wrote:
On 22.09.23 14:54, Richard W.M. Jones wrote:
On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote:
On 21.09.23 19:43, Richard W.M. Jones wrote:
So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area.
It's 2.2.0-1 from Debian (12) bookworm. I've verified that it doesn't have any distro-specific patches.
(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian would have a patches/series file in this case)
The timezone fixes are:
commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100
convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell
commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000
convert/convert_windows.ml: Handle date formats with dots instead of /
They are all included in >= 2.0
I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later.
Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way?
Or in even simpler terms, what is the time (and timezone) that this ^^^ machine was booted?
I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future.
So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*.
From what I can see, the host machine time zone is irrelevant (UTC+2).
I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?
Right, I think there is a timezone disagreement between how we format the timestamp and how schtasks.exe takes it.
What matters here is the /ST (start time) flag.
Today we have (in the common submodule):
add "$d = (get-date).AddSeconds(120)"; add "$dtfinfo = [System.Globalization.DateTimeFormatInfo]::CurrentInfo"; add "$sdp = $dtfinfo.ShortDatePattern"; add "$sdp = $sdp -replace 'y+', 'yyyy'"; add "$sdp = $sdp -replace 'M+', 'MM'"; add "$sdp = $sdp -replace 'd+', 'dd'"; add "schtasks.exe /Create /SC ONCE `"; add " /ST $d.ToString('HH:mm') /SD $d.ToString($sdp) `"; add " /RU SYSTEM /TN Firstboot-qemu-ga `"; add (sprintf " /TR \"C:\\%s /forcerestart /qn /l+*vx C:\\%s.log\"" msi_path msi_path);
Note that for the /ST option's argument, we only perform the following steps:
$d = (get-date).AddSeconds(120)
/ST $d.ToString('HH:mm')
This actually goes back to commit dc66e78fa37d ("windows: delay installation of qemu-ga MSI", 2020-03-10). The timestamp massaging we've since done only targeted the /SD (start date) option, not the start time (/ST) one!
So the problem may be that
(get-date).AddSeconds(120).ToString('HH:mm')
formats the hour:minute timestamp in UTC (why though?), but the /ST option takes hour:minute in local time.
Interestingly, DateTime objects seem to have a "Kind" property, which may be Utc, Local, or Unspec.
https://learn.microsoft.com/en-us/dotnet/api/system.datetime.kind
It seems to be used when converting from UTC to local or vice versa, and it probably influences how ToString() behaves too. I thought "get-date" returned a Local one, and /ST took a local one as well, but perhaps "get-date" returns a UTC timestamp in this case (when run from the script)?
I think I have an idea what's happening. This is the part of the XML description of the guest: <clock offset="utc"/> Which sets the time in the guest to UTC during boot. So these are the steps causing the issue I'm seeing: 1. The machine boots with the clock set to UTC. Windows assumes this is local time. 1. The above code snippet gets run in the boot process, scheduling the task to run 2 minutes into the future. 2. Within these two minutes, the Windows NTP client runs, which notices a huge offset of several hours, and sets the clock back. 3. Since in this case the timezone is UTC-7, the task is now scheduled to run in 7 hours, 2 minutes. From what I can see, this is only an issue on machines that have a timezone set with a negative offset. UTC and any positive offset end up triggering the scheduled task just fine, as any task set in the past will be triggered. I tried this with admin privileges: PS C:\Users\User> W32tm /resync The following error occurred: The service has not been started. (0x80070426) but that does not synchronize the time. Clicking on "sync now" in the date & time settings however works fine. So I don't really know how the clock is synchronized on Windows 11, but that should happen before the task is scheduled.
Laszlo
Rich.
The code we run is here:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Mon, Sep 25, 2023 at 01:09:15PM +0200, Lee Garrett wrote:
On 23.09.23 19:37, Laszlo Ersek wrote:
On 9/22/23 16:47, Lee Garrett wrote:
On 22.09.23 14:54, Richard W.M. Jones wrote:
On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote:
On 21.09.23 19:43, Richard W.M. Jones wrote: >So this is probably another instance or variation of the timezone >formatting problem (of schtasks). Which version of virt-v2v is this? >I want to check that you have a version with all the latest patches in >this area.
It's 2.2.0-1 from Debian (12) bookworm. I've verified that it doesn't have any distro-specific patches.
(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian would have a patches/series file in this case)
The timezone fixes are:
commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100
convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell
commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000
convert/convert_windows.ml: Handle date formats with dots instead of /
They are all included in >= 2.0
I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later.
Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way?
Or in even simpler terms, what is the time (and timezone) that this ^^^ machine was booted?
I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future.
So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*.
From what I can see, the host machine time zone is irrelevant (UTC+2).
I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?
Right, I think there is a timezone disagreement between how we format the timestamp and how schtasks.exe takes it.
What matters here is the /ST (start time) flag.
Today we have (in the common submodule):
add "$d = (get-date).AddSeconds(120)"; add "$dtfinfo = [System.Globalization.DateTimeFormatInfo]::CurrentInfo"; add "$sdp = $dtfinfo.ShortDatePattern"; add "$sdp = $sdp -replace 'y+', 'yyyy'"; add "$sdp = $sdp -replace 'M+', 'MM'"; add "$sdp = $sdp -replace 'd+', 'dd'"; add "schtasks.exe /Create /SC ONCE `"; add " /ST $d.ToString('HH:mm') /SD $d.ToString($sdp) `"; add " /RU SYSTEM /TN Firstboot-qemu-ga `"; add (sprintf " /TR \"C:\\%s /forcerestart /qn /l+*vx C:\\%s.log\"" msi_path msi_path);
Note that for the /ST option's argument, we only perform the following steps:
$d = (get-date).AddSeconds(120)
/ST $d.ToString('HH:mm')
This actually goes back to commit dc66e78fa37d ("windows: delay installation of qemu-ga MSI", 2020-03-10). The timestamp massaging we've since done only targeted the /SD (start date) option, not the start time (/ST) one!
So the problem may be that
(get-date).AddSeconds(120).ToString('HH:mm')
formats the hour:minute timestamp in UTC (why though?), but the /ST option takes hour:minute in local time.
Interestingly, DateTime objects seem to have a "Kind" property, which may be Utc, Local, or Unspec.
https://learn.microsoft.com/en-us/dotnet/api/system.datetime.kind
It seems to be used when converting from UTC to local or vice versa, and it probably influences how ToString() behaves too. I thought "get-date" returned a Local one, and /ST took a local one as well, but perhaps "get-date" returns a UTC timestamp in this case (when run from the script)?
I think I have an idea what's happening. This is the part of the XML description of the guest:
<clock offset="utc"/>
Dan: For virt-v2v of Windows guests do you think we should always force <clock offset="localtime"/> (but leave it at "utc" for non-Windows)? I'm not clear what exactly should be the source of truth here. I don't see anything in osinfo-db, unless I'm missing something. I also can't see anything relevant in libvirt. Rich.
Which sets the time in the guest to UTC during boot.
So these are the steps causing the issue I'm seeing: 1. The machine boots with the clock set to UTC. Windows assumes this is local time. 1. The above code snippet gets run in the boot process, scheduling the task to run 2 minutes into the future. 2. Within these two minutes, the Windows NTP client runs, which notices a huge offset of several hours, and sets the clock back. 3. Since in this case the timezone is UTC-7, the task is now scheduled to run in 7 hours, 2 minutes.
From what I can see, this is only an issue on machines that have a timezone set with a negative offset. UTC and any positive offset end up triggering the scheduled task just fine, as any task set in the past will be triggered.
I tried this with admin privileges:
PS C:\Users\User> W32tm /resync The following error occurred: The service has not been started. (0x80070426)
but that does not synchronize the time.
Clicking on "sync now" in the date & time settings however works fine. So I don't really know how the clock is synchronized on Windows 11, but that should happen before the task is scheduled.
Laszlo
Rich.
The code we run is here:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On Mon, Sep 25, 2023 at 01:14:40PM +0100, Richard W.M. Jones wrote:
On Mon, Sep 25, 2023 at 01:09:15PM +0200, Lee Garrett wrote:
On 23.09.23 19:37, Laszlo Ersek wrote:
On 9/22/23 16:47, Lee Garrett wrote:
On 22.09.23 14:54, Richard W.M. Jones wrote:
On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote:
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: >On 21.09.23 19:43, Richard W.M. Jones wrote: >>So this is probably another instance or variation of the timezone >>formatting problem (of schtasks). Which version of virt-v2v is this? >>I want to check that you have a version with all the latest patches in >>this area. > >It's 2.2.0-1 from Debian (12) bookworm. I've verified that it >doesn't have any distro-specific patches. > >(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian >would have a patches/series file in this case)
The timezone fixes are:
commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100
convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell
commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000
convert/convert_windows.ml: Handle date formats with dots instead of /
They are all included in >= 2.0
I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later.
Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
>Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way?
Or in even simpler terms, what is the time (and timezone) that this ^^^ machine was booted?
I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future.
So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*.
From what I can see, the host machine time zone is irrelevant (UTC+2).
I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?
Right, I think there is a timezone disagreement between how we format the timestamp and how schtasks.exe takes it.
What matters here is the /ST (start time) flag.
Today we have (in the common submodule):
add "$d = (get-date).AddSeconds(120)"; add "$dtfinfo = [System.Globalization.DateTimeFormatInfo]::CurrentInfo"; add "$sdp = $dtfinfo.ShortDatePattern"; add "$sdp = $sdp -replace 'y+', 'yyyy'"; add "$sdp = $sdp -replace 'M+', 'MM'"; add "$sdp = $sdp -replace 'd+', 'dd'"; add "schtasks.exe /Create /SC ONCE `"; add " /ST $d.ToString('HH:mm') /SD $d.ToString($sdp) `"; add " /RU SYSTEM /TN Firstboot-qemu-ga `"; add (sprintf " /TR \"C:\\%s /forcerestart /qn /l+*vx C:\\%s.log\"" msi_path msi_path);
Note that for the /ST option's argument, we only perform the following steps:
$d = (get-date).AddSeconds(120)
/ST $d.ToString('HH:mm')
This actually goes back to commit dc66e78fa37d ("windows: delay installation of qemu-ga MSI", 2020-03-10). The timestamp massaging we've since done only targeted the /SD (start date) option, not the start time (/ST) one!
So the problem may be that
(get-date).AddSeconds(120).ToString('HH:mm')
formats the hour:minute timestamp in UTC (why though?), but the /ST option takes hour:minute in local time.
Interestingly, DateTime objects seem to have a "Kind" property, which may be Utc, Local, or Unspec.
https://learn.microsoft.com/en-us/dotnet/api/system.datetime.kind
It seems to be used when converting from UTC to local or vice versa, and it probably influences how ToString() behaves too. I thought "get-date" returned a Local one, and /ST took a local one as well, but perhaps "get-date" returns a UTC timestamp in this case (when run from the script)?
I think I have an idea what's happening. This is the part of the XML description of the guest:
<clock offset="utc"/>
Dan:
For virt-v2v of Windows guests do you think we should always force <clock offset="localtime"/> (but leave it at "utc" for non-Windows)?
Generally 'localtime' is best for a default Windows install. IIUC, there is a way to tell Windows to assume the RTC is always in UTC, but its default behaviour is to assume localtime.
I'm not clear what exactly should be the source of truth here. I don't see anything in osinfo-db, unless I'm missing something.
Yeah, we don't record anything in this area. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Sep 25, 2023 at 01:25:06PM +0100, Daniel P. Berrangé wrote:
On Mon, Sep 25, 2023 at 01:14:40PM +0100, Richard W.M. Jones wrote:
On Mon, Sep 25, 2023 at 01:09:15PM +0200, Lee Garrett wrote:
On 23.09.23 19:37, Laszlo Ersek wrote:
On 9/22/23 16:47, Lee Garrett wrote:
On 22.09.23 14:54, Richard W.M. Jones wrote:
On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote: >On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: >>On 21.09.23 19:43, Richard W.M. Jones wrote: >>>So this is probably another instance or variation of the timezone >>>formatting problem (of schtasks). Which version of virt-v2v is this? >>>I want to check that you have a version with all the latest patches in >>>this area. >> >>It's 2.2.0-1 from Debian (12) bookworm. I've verified that it >>doesn't have any distro-specific patches. >> >>(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian >>would have a patches/series file in this case) > >The timezone fixes are: > >commit 597d177567234c3a539098c423649781424eeb6f >Author: Laszlo Ersek <lersek@redhat.com> >Date: Tue Mar 8 15:30:51 2022 +0100 > > convert_windows: rewrite "configure_qemu_ga" script purely in >PowerShell > >commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f >Author: Richard W.M. Jones <rjones@redhat.com> >Date: Fri Nov 12 08:47:55 2021 +0000 > > convert/convert_windows.ml: Handle date formats with dots >instead of / > >They are all included in >= 2.0 > >I wonder if 597d177567 has a subtle flaw, or if we introduced a bug >somewhere when refactoring this code later. > >Lee: Do you have a theory about exactly what is wrong with the >schtasks date? Like what was it supposed to be, assuming it was 120 >seconds in the future from boot time, versus what it was set to: > >>Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready > >Could a date or time field have not been swapped or been corrupted >in some predictable way?
Or in even simpler terms, what is the time (and timezone) that this ^^^ machine was booted?
I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future.
So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*.
From what I can see, the host machine time zone is irrelevant (UTC+2).
I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?
Right, I think there is a timezone disagreement between how we format the timestamp and how schtasks.exe takes it.
What matters here is the /ST (start time) flag.
Today we have (in the common submodule):
add "$d = (get-date).AddSeconds(120)"; add "$dtfinfo = [System.Globalization.DateTimeFormatInfo]::CurrentInfo"; add "$sdp = $dtfinfo.ShortDatePattern"; add "$sdp = $sdp -replace 'y+', 'yyyy'"; add "$sdp = $sdp -replace 'M+', 'MM'"; add "$sdp = $sdp -replace 'd+', 'dd'"; add "schtasks.exe /Create /SC ONCE `"; add " /ST $d.ToString('HH:mm') /SD $d.ToString($sdp) `"; add " /RU SYSTEM /TN Firstboot-qemu-ga `"; add (sprintf " /TR \"C:\\%s /forcerestart /qn /l+*vx C:\\%s.log\"" msi_path msi_path);
Note that for the /ST option's argument, we only perform the following steps:
$d = (get-date).AddSeconds(120)
/ST $d.ToString('HH:mm')
This actually goes back to commit dc66e78fa37d ("windows: delay installation of qemu-ga MSI", 2020-03-10). The timestamp massaging we've since done only targeted the /SD (start date) option, not the start time (/ST) one!
So the problem may be that
(get-date).AddSeconds(120).ToString('HH:mm')
formats the hour:minute timestamp in UTC (why though?), but the /ST option takes hour:minute in local time.
Interestingly, DateTime objects seem to have a "Kind" property, which may be Utc, Local, or Unspec.
https://learn.microsoft.com/en-us/dotnet/api/system.datetime.kind
It seems to be used when converting from UTC to local or vice versa, and it probably influences how ToString() behaves too. I thought "get-date" returned a Local one, and /ST took a local one as well, but perhaps "get-date" returns a UTC timestamp in this case (when run from the script)?
I think I have an idea what's happening. This is the part of the XML description of the guest:
<clock offset="utc"/>
Dan:
For virt-v2v of Windows guests do you think we should always force <clock offset="localtime"/> (but leave it at "utc" for non-Windows)?
Generally 'localtime' is best for a default Windows install.
IIUC, there is a way to tell Windows to assume the RTC is always in UTC, but its default behaviour is to assume localtime.
It turns out there's a Windows registry key we could check: https://wiki.archlinux.org/title/System_time#UTC_in_Microsoft_Windows According to this: https://superuser.com/questions/975717/does-windows-10-support-utc-as-bios-t... ... it seems like it's a QWORD (type 'b') rather than a DWORD for 64 bit Windows variants, but that doesn't matter as we would only need to read the registry and not modify it. The key seems to be completely missing for normal (localtime) Windows guests that I have examined. I checked virt-v2v and we don't do this right now. In fact we don't set the <clock> element at all, we just leave it up to libvirt or other tools. As this seems quite important, if someone wants to file a bug (https://issues.redhat.com/ -> select "RHEL" as the project) I can look at implementing this. Rich.
I'm not clear what exactly should be the source of truth here. I don't see anything in osinfo-db, unless I'm missing something.
Yeah, we don't record anything in this area.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Hi Richard, Sorry, I missed the email, I found out that the ova OS used by the customer is win11 uefi, so I tried to reproduce the error today using win11-uefi guest Steps: 1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with UTC-8 timezone from VMware by v2v, then check the installation status after finishing conversion Summary the test result as below: qemu-ga Win10-efi-UTC-8 PASS Win11-non-efi UTC-8 PASS Win11-efi-UTC-8 FAIL(can't find qemu-ga log in c:\) Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\) Win2022-efi-UTC-8 PASS Based on the above results, I think this problem is only caused by win11-efi and has nothing to do with the time zone "UTC-8" The following are existing bugs about qemu-ga: *Bug 1820144* <https://bugzilla.redhat.com/show_bug.cgi?id=1820144> - cannot install qemu-ga to some guests even if scheduled qemu-ga installation task exist *Bug 2114809* <https://bugzilla.redhat.com/show_bug.cgi?id=2114809> - Can't install qemu-ga because of network address error after converting MD-RAID1 win11 host by virt-p2v *Bug 1820152* <https://bugzilla.redhat.com/show_bug.cgi?id=1820152> - Fail to execute installation-qemu-ga.msi for win2012r2 and win8.1-i386 guests Thanks & Regards Ming Xie On Fri, Sep 22, 2023 at 6:40 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote:
On 21.09.23 19:43, Richard W.M. Jones wrote:
So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area.
It's 2.2.0-1 from Debian (12) bookworm. I've verified that it doesn't have any distro-specific patches.
( https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian would have a patches/series file in this case)
The timezone fixes are:
commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100
convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell
commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000
convert/convert_windows.ml: Handle date formats with dots instead of /
They are all included in >= 2.0
I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later.
Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way?
The code we run is here:
https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7...
Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Wed, Sep 27, 2023 at 08:46:58PM +0800, Ming Xie wrote:
Hi Richard,
Sorry, I missed the email, I found out that the ova OS used by the customer is win11 uefi, so I tried to reproduce the error today using win11-uefi guest
Steps: 1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with UTC-8 timezone from VMware by v2v, then check the installation status after finishing conversion
Summary the test result as below:
qemu-ga
Win10-efi-UTC-8 PASS Win11-non-efi UTC-8 PASS Win11-efi-UTC-8 FAIL(can't find qemu-ga log in c:\) Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\) Win2022-efi-UTC-8 PASS
Thanks for testing. The symptoms look the same as Lee reported. You're running virt-v2v from RHEL 9? If so I will prepare a scratch RHEL 9 package with the patches which we think might fix this, for testing. Rich.
Based on the above results, I think this problem is only caused by win11-efi and has nothing to do with the time zone "UTC-8"
The following are existing bugs about qemu-ga:
Bug 1820144 - cannot install qemu-ga to some guests even if scheduled qemu-ga installation task exist Bug 2114809 - Can't install qemu-ga because of network address error after converting MD-RAID1 win11 host by virt-p2v Bug 1820152 - Fail to execute installation-qemu-ga.msi for win2012r2 and win8.1-i386 guests
Thanks & Regards Ming Xie
On Fri, Sep 22, 2023 at 6:40 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: > On 21.09.23 19:43, Richard W.M. Jones wrote: > >So this is probably another instance or variation of the timezone > >formatting problem (of schtasks). Which version of virt-v2v is this? > >I want to check that you have a version with all the latest patches in > >this area. > > It's 2.2.0-1 from Debian (12) bookworm. I've verified that it > doesn't have any distro-specific patches. > > (https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/ debian > would have a patches/series file in this case)
The timezone fixes are:
commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek@redhat.com> Date: Tue Mar 8 15:30:51 2022 +0100
convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell
commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones@redhat.com> Date: Fri Nov 12 08:47:55 2021 +0000
convert/convert_windows.ml: Handle date formats with dots instead of /
They are all included in >= 2.0
I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later.
Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to:
> Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready
Could a date or time field have not been swapped or been corrupted in some predictable way?
The code we run is here:
https://github.com/libguestfs/libguestfs-common/blob/ e70d89a58dae068be2e19c7c21558707261af96a/mlcustomize/inject_virtio_win.ml# L571
Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/ ~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
Hi Richard, On Thu, Sep 28, 2023 at 1:02 AM Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Sep 27, 2023 at 08:46:58PM +0800, Ming Xie wrote:
Hi Richard,
Sorry, I missed the email, I found out that the ova OS used by the customer is win11 uefi, so I tried to reproduce the error today using win11-uefi guest
Steps: 1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with UTC-8 timezone from VMware by v2v, then check the installation status after finishing conversion
Summary the test result as below:
qemu-ga
Win10-efi-UTC-8 PASS Win11-non-efi UTC-8 PASS Win11-efi-UTC-8 FAIL(can't find qemu-ga log in c:\) Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\) Win2022-efi-UTC-8 PASS
Thanks for testing. The symptoms look the same as Lee reported.
You're running virt-v2v from RHEL 9? If so I will prepare a scratch RHEL 9 package with the patches which we think might fix this, for testing.
Yes, I reproduced the problem using rhel9 virt-v2v, because I saw that the customer was also using rhel9 virt-v2v (version: 2.2), I saw that you wanted to solve the problem by fixing the date of the schtask command, but I see that the schtask command used to install qemu-ga has been changed to the one shown in "qemu-ga-command-rhel9.png" on rhel9 Besides, qemu-ga can be installed successfully after converting win11-efi -UTC-8 guest on rhel8.9, which means I cannot reproduce the issue on rhel8 and the schtask command used for installing qemu-ga is old one as '' qemu-ga-command-rhel8.png' ' shown on rhel8.9, so I'm not sure if the fix can be work with rhel9 virt-v2v Regards Ming Xie
On Thu, Sep 28, 2023 at 04:03:33PM +0800, Ming Xie wrote:
Yes, I reproduced the problem using rhel9 virt-v2v, because I saw that the customer was also using rhel9 virt-v2v (version: 2.2), I saw that you wanted to solve the problem by fixing the date of the schtask command, but I see that the schtask command used to install qemu-ga has been changed to the one shown in "qemu-ga-command-rhel9.png" on rhel9
The \windows\temp\install-qemu-ga-x86_64.msi.ps1 script will contain the actual install commands. I'll send you an updated v2v package anyway. Rich.
Besides, qemu-ga can be installed successfully after converting win11-efi -UTC-8 guest on rhel8.9, which means I cannot reproduce the issue on rhel8 and the schtask command used for installing qemu-ga is old one as '' qemu-ga-command-rhel8.png' ' shown on rhel8.9, so I'm not sure if the fix can be work with rhel9 virt-v2v
Regards Ming Xie
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Ming, For some reason the build seems "stuck" at the moment, but if it finishes could you see if this fixes the RHEL 9 case? https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547 Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
Hi Richard, On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones <rjones@redhat.com> wrote:
Ming,
For some reason the build seems "stuck" at the moment, but if it finishes could you see if this fixes the RHEL 9 case?
https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547
Sure, no problem, but I don't see the package available now, I will check later. BTW, is there a bug for this issue? Regards Ming Xie
On Thu, Sep 28, 2023 at 07:41:55PM +0800, Ming Xie wrote:
Hi Richard,
On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones <rjones@redhat.com> wrote:
Ming,
For some reason the build seems "stuck" at the moment, but if it finishes could you see if this fixes the RHEL 9 case?
https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547
Sure, no problem, but I don't see the package available now, I will check later.
The build seems stuck. I asked on the RHEL mailing list if someone can look at it.
BTW, is there a bug for this issue?
Not at the moment. Rich.
Regards Ming Xie
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Hi Richard, On Thu, Sep 28, 2023 at 7:54 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, Sep 28, 2023 at 07:41:55PM +0800, Ming Xie wrote:
Hi Richard,
On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones <rjones@redhat.com> wrote:
Ming,
For some reason the build seems "stuck" at the moment, but if it finishes could you see if this fixes the RHEL 9 case?
https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547
Sure, no problem, but I don't see the package available now, I will check later.
The build seems stuck. I asked on the RHEL mailing list if someone can look at it.
BTW, is there a bug for this issue?
Not at the moment.
I filed a bug https://issues.redhat.com/browse/RHEL-12104 to track the issue in downstream, thanks! Regards Ming Xie
On Thu, Sep 28, 2023 at 07:41:55PM +0800, Ming Xie wrote:
Hi Richard,
On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones <rjones@redhat.com> wrote:
Ming,
For some reason the build seems "stuck" at the moment, but if it finishes could you see if this fixes the RHEL 9 case?
https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547
It finished building now. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
Hi Richard, On Fri, Sep 29, 2023 at 5:45 AM Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, Sep 28, 2023 at 07:41:55PM +0800, Ming Xie wrote:
Hi Richard,
On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones <rjones@redhat.com> wrote:
Ming,
For some reason the build seems "stuck" at the moment, but if it finishes could you see if this fixes the RHEL 9 case?
https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547
It finished building now.
Use virt-v2v-2.3.4-6.qemugatzbug1.el9.x86_64 to test and get the following results, the problem has not been fixed yet. qemu-ga win10-non-UTC-8 PASS Win10-efi-UTC-8 FAIL Win11-non-efi UTC-8 PASS Win11-efi-UTC-8 FAIL(can't find qemu-ga log in c:\) Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\) Win2019-efi-UTC-8 PASS Win2022-efi-UTC-8 PASS Regards Ming Xie
On Mon, Oct 02, 2023 at 01:00:00PM +0800, Ming Xie wrote:
Hi Richard,
On Fri, Sep 29, 2023 at 5:45 AM Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, Sep 28, 2023 at 07:41:55PM +0800, Ming Xie wrote: > Hi Richard, > > On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > Ming, > > For some reason the build seems "stuck" at the moment, but > if it finishes could you see if this fixes the RHEL 9 case? > > https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547
It finished building now.
Use virt-v2v-2.3.4-6.qemugatzbug1.el9.x86_64 to test and get the following results, the problem has not been fixed yet.
qemu-ga win10-non-UTC-8 PASS Win10-efi-UTC-8 FAIL Win11-non-efi UTC-8 PASS Win11-efi-UTC-8 FAIL(can't find qemu-ga log in c:\) Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\) Win2019-efi-UTC-8 PASS Win2022-efi-UTC-8 PASS
OK thanks for testing it. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Hi Ming, On 27.09.23 14:46, Ming Xie wrote:
Hi Richard,
Sorry, I missed the email, I found out that the ova OS used by the customer is win11 uefi, so I tried to reproduce the error today using win11-uefi guest
Steps: 1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with UTC-8 timezone from VMware by v2v, then check the installation status after finishing conversion
Summary the test result as below:
qemu-ga
Win10-efi-UTC-8 PASS Win11-non-efi UTC-8 PASS Win11-efi-UTC-8 FAIL(can't find qemu-ga log in c:\) Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\) Win2022-efi-UTC-8 PASS
Based on the above results, I think this problem is only caused by win11-efi and has nothing to do with the time zone "UTC-8"
hmm, where are you exactly setting the timezone? Within the guest, the scheduled task, or in the libvirt XML description? The Windows 11 trial VM in question (downloaded from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ -> https://aka.ms/windev_VM_vmware) comes with the timezone within the VM set to US/Pacific. If I also set the RTC clock for the VM in the libvirt XML description after conversion like this: <clock offset="timezone" timezone="US/Pacific"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock> Then qemu-ga successfully gets installed after a 2 minute delay. If I leave the default libvirt XML description at: <clock offset="utc"/> then the issue of qemu-ga not installing remains (the task is scheduled to run 7 hours in the future). So it is clearly an issue of the timezone of the guest and what is passed from libvirt to be out of sync. Would be nice to compare how you test it so we're on the same page. (Also, to clarify, I'm running virt-v2v 2.2.0 without the proposed patches)
The following are existing bugs about qemu-ga:
*Bug 1820144* <https://bugzilla.redhat.com/show_bug.cgi?id=1820144> - cannot install qemu-ga to some guests even if scheduled qemu-ga installation task exist *Bug 2114809* <https://bugzilla.redhat.com/show_bug.cgi?id=2114809> - Can't install qemu-ga because of network address error after converting MD-RAID1 win11 host by virt-p2v *Bug 1820152* <https://bugzilla.redhat.com/show_bug.cgi?id=1820152> - Fail to execute installation-qemu-ga.msi for win2012r2 and win8.1-i386 guests
Thanks & Regards Ming Xie
Greetings, Lee
(resend, as it bounced on tyan-all@redhat.com) Hi Ming, On 27.09.23 14:46, Ming Xie wrote:
Hi Richard,
Sorry, I missed the email, I found out that the ova OS used by the customer is win11 uefi, so I tried to reproduce the error today using win11-uefi guest
Steps: 1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with UTC-8 timezone from VMware by v2v, then check the installation status after finishing conversion
Summary the test result as below:
qemu-ga
Win10-efi-UTC-8 PASS Win11-non-efi UTC-8 PASS Win11-efi-UTC-8 FAIL(can't find qemu-ga log in c:\) Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\) Win2022-efi-UTC-8 PASS
Based on the above results, I think this problem is only caused by win11-efi and has nothing to do with the time zone "UTC-8"
hmm, where are you exactly setting the timezone? Within the guest, the scheduled task, or in the libvirt XML description? The Windows 11 trial VM in question (downloaded from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ -> https://aka.ms/windev_VM_vmware) comes with the timezone within the VM set to US/Pacific. If I also set the RTC clock for the VM in the libvirt XML description after conversion like this: <clock offset="timezone" timezone="US/Pacific"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock> Then qemu-ga successfully gets installed after a 2 minute delay. If I leave the default libvirt XML description at: <clock offset="utc"/> then the issue of qemu-ga not installing remains (the task is scheduled to run 7 hours in the future). So it is clearly an issue of the timezone of the guest and what is passed from libvirt to be out of sync. Would be nice to compare how you test it so we're on the same page. (Also, to clarify, I'm running virt-v2v 2.2.0 without the proposed patches)
The following are existing bugs about qemu-ga:
*Bug 1820144* <https://bugzilla.redhat.com/show_bug.cgi?id=1820144> - cannot install qemu-ga to some guests even if scheduled qemu-ga installation task exist *Bug 2114809* <https://bugzilla.redhat.com/show_bug.cgi?id=2114809> - Can't install qemu-ga because of network address error after converting MD-RAID1 win11 host by virt-p2v *Bug 1820152* <https://bugzilla.redhat.com/show_bug.cgi?id=1820152> - Fail to execute installation-qemu-ga.msi for win2012r2 and win8.1-i386 guests
Thanks & Regards Ming Xie
Greetings, Lee
Hi Lee, On Thu, Sep 28, 2023 at 10:55 PM Lee Garrett <lgarrett@rocketjump.eu> wrote:
(resend, as it bounced on tyan-all@redhat.com)
Hi Ming,
On 27.09.23 14:46, Ming Xie wrote:
Hi Richard,
Sorry, I missed the email, I found out that the ova OS used by the customer is win11 uefi, so I tried to reproduce the error today using win11-uefi guest
Steps: 1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with UTC-8 timezone from VMware by v2v, then check the installation status after finishing conversion
Summary the test result as below:
qemu-ga
Win10-efi-UTC-8 PASS Win11-non-efi UTC-8 PASS Win11-efi-UTC-8 FAIL(can't find qemu-ga log in c:\) Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\) Win2022-efi-UTC-8 PASS
Based on the above results, I think this problem is only caused by win11-efi and has nothing to do with the time zone "UTC-8"
hmm, where are you exactly setting the timezone? Within the guest, the scheduled task, or in the libvirt XML description?
The timezone is UTC-8(US&Canada Pacific Time) in third win11-efi and the clock is UTC after v2v conversion
The Windows 11 trial VM in question (downloaded from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ -> https://aka.ms/windev_VM_vmware) comes with the timezone within the VM set to US/Pacific.
If I also set the RTC clock for the VM in the libvirt XML description after conversion like this: <clock offset="timezone" timezone="US/Pacific"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock>
Then qemu-ga successfully gets installed after a 2 minute delay. If I leave the default libvirt XML description at:
<clock offset="utc"/>
This method does not work in my win11-efi guest, the timezone in the guest is UTC-8(US&Canada Pacific Time) ) and changed the clock as above configuration，restarted the guest but qemu-ga still can‘t install successfully Regards Ming Xie
participants (5)
-
Daniel P. Berrangé -
Laszlo Ersek -
Lee Garrett -
Ming Xie -
Richard W.M. Jones