On Sun, Sep 10, 2023 at 11:34:22AM +0100, Richard W.M. Jones wrote: > This message got caught in moderation because the attachment > was large. I put the attachment here: > > http://oirase.annexia.org/tmp/2023-lgarrett-virt-v2v-debug-output.txt > > ----- Forwarded message from Lee Garrett <lgarrett(a)rocketjump.eu&gt; ----- > > Date: Sun, 10 Sep 2023 00:09:17 +0200 > From: Lee Garrett <lgarrett(a)rocketjump.eu&gt; > To: libguestfs(a)redhat.com > Subject: virt-v2v creating image that does not install guest agent on first > boot > > Hi, > > rwmjones from #guestfs on libera IRC pointed me to this mail address. > > I've noticed that converting the latest Windows 11 trial image via > virt-v2v does not install the guest agent any more on first boot. I > have let the machine settle for ~10 minutes to ensure it's not just > because I'm too impatient. I can see C:\qemu-ga-x86-64.msi, and also a > bunch of files in "C:\Program Files\Guestfs\Firstboot\" and > .\Scripts, so the files are definitely copied onto the VM image, just > not executed for some reason. I can see from the log that these are installed in the image. > The vmware image was downloaded from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/. > The issue should be 100% reproducible (at least for me). I recall that > an older image from two, three weeks ago downloaded from the same > location did not have this issue. There guest agent was installed and > working on the first boot. Unfortunately I overwrote the image so I > cannot verify this. > > I have attached the debug output of the virt-v2v run of the latest > image where this issue is apparent (guest agent not being installed). > > Please let me know if there's anything else I can check, or if you > need any further info. The main thing to look for is any debugging output which should appear in c:\program files\guestfs\firstboot\log.txt It would be interesting to know if this file is present at all, and what exactly it contains.

I've CC'd my colleague Ming on this to see if she has noticed anything with recent Windows 11. Rich. > Cheers, > Lee > > ----- End forwarded message ----- > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > libguestfs lets you edit virtual machines. Supports shell scripting, > bindings from many languages. http://libguestfs.org > _______________________________________________ > Libguestfs mailing list > Libguestfs(a)redhat.com > https://listman.redhat.com/mailman/listinfo/libguestfs

If we suspect that the whole firstboot mechanism might not be working with the new version of Windows, one way to test it (on this one, or a freshly installed Windows VM) would be: $ virt-customize -a windows.img --firstboot-command 'echo hello' and see if "hello" is written in some form to the log.txt file inside the guest after it boots. If that doesn't work then it's likely some change in Windows which is breaking firstboot support. Rich.

On Thu, Sep 21, 2023 at 04:50:26PM +0200, Lee Garrett wrote: > After testing it again today, the guest agent is still not > installed, however this time the C:\Program > Files\Guestfs\Firstboot\log.txt exists: > > starting firstboot service > running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0001-wait-pnp.bat" > 1 file(s) moved. > Wait for PnP to complete > .... exit code 0 > running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat" > 1 file(s) moved. > Removing any previously scheduled qemu-ga installation > ERROR: The system cannot find the file specified. > Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi > SUCCESS: The scheduled task "Firstboot-qemu-ga" has successfully been created. > .... exit code 0 > running "C:\Program > Files\Guestfs\Firstboot\scripts\5000-0003-uninstall-VMware-Tools.bat" > 1 file(s) moved. > uninstalling VMware Tools > .... exit code 0 > uninstalling firstboot service > Service uninstalled successfully > > > The folder scripts is empty, all scripts are in scripts-done. The > contents of the folder are: > 5000-0001-wait-pnp.bat > 123 5000-0001-wait-pnp.log > 5000-0002-install-qemu-ga-x86_64-msi-ps1.bat > 5000-0003-uninstall-VMware-Tools.bat > 5000-0003-uninstall-VMware-Tools.log > > Note that the .log is missing for qemu-ga. This is what we'd expect, except maybe the missing log for qemu-ga. It's a little odd that it is missing, but it's also possible that the log file doesn't get written if there's no output. > At least today I could twice provision the windows 11 VM, and in > both cases the main log.txt was generated by the time I checked. How > are the services scheduled? Could it be that they were delayed by > > 1h? Yes, we do actually try to delay qemu-ga installation. It should only be delayed for about 120 seconds (not 1 hour). https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e1... However there have been timezone bugs with this in the past (and maybe there still are), see the sordid history in these two links: https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e1...

Which TZ are you in?

Rich. > On 11.09.23 21:29, Richard W.M. Jones wrote: >> >> If we suspect that the whole firstboot mechanism might not be working >> with the new version of Windows, one way to test it (on this one, or a >> freshly installed Windows VM) would be: >> >> $ virt-customize -a windows.img --firstboot-command 'echo hello' >> >> and see if "hello" is written in some form to the log.txt file inside >> the guest after it boots. >> >> If that doesn't work then it's likely some change in Windows which is >> breaking firstboot support. >> >> Rich. >> > > Regards, Lee

On Thu, Sep 21, 2023 at 06:24:40PM +0200, Lee Garrett wrote: > On 21.09.23 18:08, Richard W.M. Jones wrote: >> On Thu, Sep 21, 2023 at 04:50:26PM +0200, Lee Garrett wrote: >>> After testing it again today, the guest agent is still not >>> installed, however this time the C:\Program >>> Files\Guestfs\Firstboot\log.txt exists: >>> >>> starting firstboot service >>> running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0001-wait-pnp.bat" >>> 1 file(s) moved. >>> Wait for PnP to complete >>> .... exit code 0 >>> running "C:\Program Files\Guestfs\Firstboot\scripts\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat" >>> 1 file(s) moved. >>> Removing any previously scheduled qemu-ga installation >>> ERROR: The system cannot find the file specified. >>> Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi >>> SUCCESS: The scheduled task "Firstboot-qemu-ga" has successfully been created. >>> .... exit code 0 >>> running "C:\Program >>> Files\Guestfs\Firstboot\scripts\5000-0003-uninstall-VMware-Tools.bat" >>> 1 file(s) moved. >>> uninstalling VMware Tools >>> .... exit code 0 >>> uninstalling firstboot service >>> Service uninstalled successfully >>> >>> >>> The folder scripts is empty, all scripts are in scripts-done. The >>> contents of the folder are: >>> 5000-0001-wait-pnp.bat >>> 123 5000-0001-wait-pnp.log >>> 5000-0002-install-qemu-ga-x86_64-msi-ps1.bat >>> 5000-0003-uninstall-VMware-Tools.bat >>> 5000-0003-uninstall-VMware-Tools.log >>> >>> Note that the .log is missing for qemu-ga. >> >> This is what we'd expect, except maybe the missing log for qemu-ga. >> It's a little odd that it is missing, but it's also possible that the >> log file doesn't get written if there's no output. >> >>> At least today I could twice provision the windows 11 VM, and in >>> both cases the main log.txt was generated by the time I checked. How >>> are the services scheduled? Could it be that they were delayed by > >>> 1h? >> >> Yes, we do actually try to delay qemu-ga installation. It should only >> be delayed for about 120 seconds (not 1 hour). >> >> https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e1... >> >> However there have been timezone bugs with this in the past (and maybe >> there still are), see the sordid history in these two links: >> >> https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e1... > > I see. That's not reassuring. :( > >> >> Which TZ are you in? > > The host machine is UTC+2 (Berlin summer time), however the Windows > 11 trial vmware image itself defaults to UTC-8 (Pacific Time US & > Canada). > > After a little bit of digging this indeed seems to be the issue. > > `schtasks.exe /Query` gives: > > PS C:\Users\User> schtasks.exe /Query > > Folder: \ > TaskName Next Run Time Status > ======================================== ====================== =============== > Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready > [...] > > which is ~7 hours in the future. Btw, running > `.\5000-0002-install-qemu-ga-x86_64-msi-ps1.bat` by hand works just > fine. So it definitely is just the scheduling issue that's > preventing it from running. So this is probably another instance or variation of the timezone formatting problem (of schtasks). Which version of virt-v2v is this? I want to check that you have a version with all the latest patches in this area.

Rich. >> >> Rich. >> >>> On 11.09.23 21:29, Richard W.M. Jones wrote: >>>> >>>> If we suspect that the whole firstboot mechanism might not be working >>>> with the new version of Windows, one way to test it (on this one, or a >>>> freshly installed Windows VM) would be: >>>> >>>> $ virt-customize -a windows.img --firstboot-command 'echo hello' >>>> >>>> and see if "hello" is written in some form to the log.txt file inside >>>> the guest after it boots. >>>> >>>> If that doesn't work then it's likely some change in Windows which is >>>> breaking firstboot support. >>>> >>>> Rich. >>>> >>> >>> Regards, Lee >> > > Regards, Lee

On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: > On 21.09.23 19:43, Richard W.M. Jones wrote: > >So this is probably another instance or variation of the timezone > >formatting problem (of schtasks). Which version of virt-v2v is this? > >I want to check that you have a version with all the latest patches in > >this area. > > It's 2.2.0-1 from Debian (12) bookworm. I've verified that it > doesn't have any distro-specific patches. > > (https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian > would have a patches/series file in this case) The timezone fixes are: commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek(a)redhat.com&gt; Date: Tue Mar 8 15:30:51 2022 +0100 convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones(a)redhat.com&gt; Date: Fri Nov 12 08:47:55 2021 +0000 convert/convert_windows.ml: Handle date formats with dots instead of / They are all included in >= 2.0 I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later. Lee: Do you have a theory about exactly what is wrong with the schtasks date? Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to: > Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready Could a date or time field have not been swapped or been corrupted in some predictable way?

The code we run is here: https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e1... Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

On 22.09.23 14:54, Richard W.M. Jones wrote: > On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote: >> On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: >>> On 21.09.23 19:43, Richard W.M. Jones wrote: >>>> So this is probably another instance or variation of the timezone >>>> formatting problem (of schtasks).  Which version of virt-v2v is this? >>>> I want to check that you have a version with all the latest patches in >>>> this area. >>> >>> It's 2.2.0-1 from Debian (12) bookworm. I've verified that it >>> doesn't have any distro-specific patches. >>> >>> (https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian >>> would have a patches/series file in this case) >> >> The timezone fixes are: >> >> commit 597d177567234c3a539098c423649781424eeb6f >> Author: Laszlo Ersek <lersek(a)redhat.com&gt; >> Date:   Tue Mar 8 15:30:51 2022 +0100 >> >>      convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell >> >> commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f >> Author: Richard W.M. Jones <rjones(a)redhat.com&gt; >> Date:   Fri Nov 12 08:47:55 2021 +0000 >> >>      convert/convert_windows.ml: Handle date formats with dots instead of / >> >> They are all included in >= 2.0 >> >> I wonder if 597d177567 has a subtle flaw, or if we introduced a bug >> somewhere when refactoring this code later. >> >> Lee: Do you have a theory about exactly what is wrong with the >> schtasks date?  Like what was it supposed to be, assuming it was 120 >> seconds in the future from boot time, versus what it was set to: >> >>> Firstboot-qemu-ga                        9/21/2023 4:04:00 PM   Ready >> >> Could a date or time field have not been swapped or been corrupted >> in some predictable way? > > Or in even simpler terms, what is the time (and timezone) that > this ^^^ machine was booted? I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future. So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*. From what I can see, the host machine time zone is irrelevant (UTC+2). I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?

> > Rich. > >> The code we run is here: >> >> https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e1... >> >> Ming: this could be a bug affecting PST (UTC-8) guests, perhaps >> somehow related to having a single digit month field? >> >> Rich. >> >> -- >> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones >> Read my programming and virtualization blog: http://rwmj.wordpress.com >> libguestfs lets you edit virtual machines.  Supports shell scripting, >> bindings from many languages.  http://libguestfs.org >

On 22.09.23 14:54, Richard W.M. Jones wrote: > On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote: >> On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: >>> On 21.09.23 19:43, Richard W.M. Jones wrote: >>>> So this is probably another instance or variation of the timezone >>>> formatting problem (of schtasks).  Which version of virt-v2v is this? >>>> I want to check that you have a version with all the latest patches in >>>> this area. >>> >>> It's 2.2.0-1 from Debian (12) bookworm. I've verified that it >>> doesn't have any distro-specific patches. >>> >>> (https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian >>> would have a patches/series file in this case) >> >> The timezone fixes are: >> >> commit 597d177567234c3a539098c423649781424eeb6f >> Author: Laszlo Ersek <lersek(a)redhat.com&gt; >> Date:   Tue Mar 8 15:30:51 2022 +0100 >> >>      convert_windows: rewrite "configure_qemu_ga" script purely in >> PowerShell >> >> commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f >> Author: Richard W.M. Jones <rjones(a)redhat.com&gt; >> Date:   Fri Nov 12 08:47:55 2021 +0000 >> >>      convert/convert_windows.ml: Handle date formats with dots >> instead of / >> >> They are all included in >= 2.0 >> >> I wonder if 597d177567 has a subtle flaw, or if we introduced a bug >> somewhere when refactoring this code later. >> >> Lee: Do you have a theory about exactly what is wrong with the >> schtasks date?  Like what was it supposed to be, assuming it was 120 >> seconds in the future from boot time, versus what it was set to: >> >>> Firstboot-qemu-ga                        9/21/2023 4:04:00 PM   Ready >> >> Could a date or time field have not been swapped or been corrupted >> in some predictable way? > > Or in even simpler terms, what is the time (and timezone) that > this ^^^ machine was booted? I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future. So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*. From what I can see, the host machine time zone is irrelevant (UTC+2). I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?

> > Rich. > >> The code we run is here: >> >> https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e1... >> >> Ming: this could be a bug affecting PST (UTC-8) guests, perhaps >> somehow related to having a single digit month field? >> >> Rich. >> >> --  >> Richard Jones, Virtualization Group, Red Hat >> http://people.redhat.com/~rjones >> Read my programming and virtualization blog: http://rwmj.wordpress.com >> libguestfs lets you edit virtual machines.  Supports shell scripting, >> bindings from many languages.  http://libguestfs.org >

On 23.09.23 19:37, Laszlo Ersek wrote: >On 9/22/23 16:47, Lee Garrett wrote: >>On 22.09.23 14:54, Richard W.M. Jones wrote: >>>On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote: >>>>On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: >>>>>On 21.09.23 19:43, Richard W.M. Jones wrote: >>>>>>So this is probably another instance or variation of the timezone >>>>>>formatting problem (of schtasks).  Which version of virt-v2v is this? >>>>>>I want to check that you have a version with all the latest patches in >>>>>>this area. >>>>> >>>>>It's 2.2.0-1 from Debian (12) bookworm. I've verified that it >>>>>doesn't have any distro-specific patches. >>>>> >>>>>(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian >>>>>would have a patches/series file in this case) >>>> >>>>The timezone fixes are: >>>> >>>>commit 597d177567234c3a539098c423649781424eeb6f >>>>Author: Laszlo Ersek <lersek(a)redhat.com&gt; >>>>Date:   Tue Mar 8 15:30:51 2022 +0100 >>>> >>>>      convert_windows: rewrite "configure_qemu_ga" script purely in >>>>PowerShell >>>> >>>>commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f >>>>Author: Richard W.M. Jones <rjones(a)redhat.com&gt; >>>>Date:   Fri Nov 12 08:47:55 2021 +0000 >>>> >>>>      convert/convert_windows.ml: Handle date formats with dots >>>>instead of / >>>> >>>>They are all included in >= 2.0 >>>> >>>>I wonder if 597d177567 has a subtle flaw, or if we introduced a bug >>>>somewhere when refactoring this code later. >>>> >>>>Lee: Do you have a theory about exactly what is wrong with the >>>>schtasks date?  Like what was it supposed to be, assuming it was 120 >>>>seconds in the future from boot time, versus what it was set to: >>>> >>>>>Firstboot-qemu-ga                        9/21/2023 4:04:00 PM   Ready >>>> >>>>Could a date or time field have not been swapped or been corrupted >>>>in some predictable way? >>> >>>Or in even simpler terms, what is the time (and timezone) that >>>this ^^^ machine was booted? >> >>I believe I have it figured out. >>The guest local time is currently 7:08 AM (a few minutes after >>firstboot/provisioning), pacific daylight time (UTC-7, though Windows >>displays it as "UTC-08:00"). This is the timezone that the guest comes >>configured with at first boot. The task is scheduled for 2:01 PM, >>meaning it's scheduled to run ~7 hours in the future. >> >>So it seems like the task was meant to be scheduled for 2:01 PM UTC (= >>7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*. >> >> From what I can see, the host machine time zone is irrelevant (UTC+2). >> >>I don't know where the timezone mixup comes from, though. Running >>`(get-date)` in the powershell at this point correctly returns the local >>time (7:08 AM). I guess during injection the time is in UTC, and >>schtasks.exe has no awareness of timezones? > >Right, I think there is a timezone disagreement between how we format the timestamp and how schtasks.exe takes it. > >What matters here is the /ST (start time) flag. > >Today we have (in the common submodule): > > add "$d = (get-date).AddSeconds(120)"; > add "$dtfinfo = [System.Globalization.DateTimeFormatInfo]::CurrentInfo"; > add "$sdp = $dtfinfo.ShortDatePattern"; > add "$sdp = $sdp -replace 'y+', 'yyyy'"; > add "$sdp = $sdp -replace 'M+', 'MM'"; > add "$sdp = $sdp -replace 'd+', 'dd'"; > add "schtasks.exe /Create /SC ONCE `"; > add " /ST $d.ToString('HH:mm') /SD $d.ToString($sdp) `"; > add " /RU SYSTEM /TN Firstboot-qemu-ga `"; > add (sprintf " /TR \"C:\\%s /forcerestart /qn /l+*vx C:\\%s.log\"" > msi_path msi_path); > >Note that for the /ST option's argument, we only perform the following steps: > > $d = (get-date).AddSeconds(120) > > /ST $d.ToString('HH:mm') > >This actually goes back to commit dc66e78fa37d ("windows: delay installation of qemu-ga MSI", 2020-03-10). The timestamp massaging we've since done only targeted the /SD (start date) option, not the start time (/ST) one! > >So the problem may be that > > (get-date).AddSeconds(120).ToString('HH:mm') > >formats the hour:minute timestamp in UTC (why though?), but the /ST option takes hour:minute in local time. > >Interestingly, DateTime objects seem to have a "Kind" property, which may be Utc, Local, or Unspec. > >https://learn.microsoft.com/en-us/dotnet/api/system.datetime.kind > >It seems to be used when converting from UTC to local or vice versa, and it probably influences how ToString() behaves too. I thought "get-date" returned a Local one, and /ST took a local one as well, but perhaps "get-date" returns a UTC timestamp in this case (when run from the script)? I think I have an idea what's happening. This is the part of the XML description of the guest: <clock offset="utc"/>

Which sets the time in the guest to UTC during boot. So these are the steps causing the issue I'm seeing: 1. The machine boots with the clock set to UTC. Windows assumes this is local time. 1. The above code snippet gets run in the boot process, scheduling the task to run 2 minutes into the future. 2. Within these two minutes, the Windows NTP client runs, which notices a huge offset of several hours, and sets the clock back. 3. Since in this case the timezone is UTC-7, the task is now scheduled to run in 7 hours, 2 minutes. From what I can see, this is only an issue on machines that have a timezone set with a negative offset. UTC and any positive offset end up triggering the scheduled task just fine, as any task set in the past will be triggered. I tried this with admin privileges: PS C:\Users\User> W32tm /resync The following error occurred: The service has not been started. (0x80070426) but that does not synchronize the time. Clicking on "sync now" in the date & time settings however works fine. So I don't really know how the clock is synchronized on Windows 11, but that should happen before the task is scheduled. > >Laszlo > >> >>> >>>Rich. >>> >>>>The code we run is here: >>>> >>>>https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7c21558707261af96a/mlcustomize/inject_virtio_win.ml#L571 >>>> >>>>Ming: this could be a bug affecting PST (UTC-8) guests, perhaps >>>>somehow related to having a single digit month field? >>>> >>>>Rich. >>>> >>>>-- >>>>Richard Jones, Virtualization Group, Red Hat >>>>http://people.redhat.com/~rjones >>>>Read my programming and virtualization blog: http://rwmj.wordpress.com >>>>libguestfs lets you edit virtual machines.  Supports shell scripting, >>>>bindings from many languages.  http://libguestfs.org >>> >> >

On Mon, Sep 25, 2023 at 01:14:40PM +0100, Richard W.M. Jones wrote: > On Mon, Sep 25, 2023 at 01:09:15PM +0200, Lee Garrett wrote: > > On 23.09.23 19:37, Laszlo Ersek wrote: > > >On 9/22/23 16:47, Lee Garrett wrote: > > >>On 22.09.23 14:54, Richard W.M. Jones wrote: > > >>>On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote: > > >>>>On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: > > >>>>>On 21.09.23 19:43, Richard W.M. Jones wrote: > > >>>>>>So this is probably another instance or variation of the timezone > > >>>>>>formatting problem (of schtasks).  Which version of virt-v2v is this? > > >>>>>>I want to check that you have a version with all the latest patches in > > >>>>>>this area. > > >>>>> > > >>>>>It's 2.2.0-1 from Debian (12) bookworm. I've verified that it > > >>>>>doesn't have any distro-specific patches. > > >>>>> > > >>>>>(https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian > > >>>>>would have a patches/series file in this case) > > >>>> > > >>>>The timezone fixes are: > > >>>> > > >>>>commit 597d177567234c3a539098c423649781424eeb6f > > >>>>Author: Laszlo Ersek <lersek(a)redhat.com&gt; > > >>>>Date:   Tue Mar 8 15:30:51 2022 +0100 > > >>>> > > >>>>      convert_windows: rewrite "configure_qemu_ga" script purely in > > >>>>PowerShell > > >>>> > > >>>>commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f > > >>>>Author: Richard W.M. Jones <rjones(a)redhat.com&gt; > > >>>>Date:   Fri Nov 12 08:47:55 2021 +0000 > > >>>> > > >>>>      convert/convert_windows.ml: Handle date formats with dots > > >>>>instead of / > > >>>> > > >>>>They are all included in >= 2.0 > > >>>> > > >>>>I wonder if 597d177567 has a subtle flaw, or if we introduced a bug > > >>>>somewhere when refactoring this code later. > > >>>> > > >>>>Lee: Do you have a theory about exactly what is wrong with the > > >>>>schtasks date?  Like what was it supposed to be, assuming it was 120 > > >>>>seconds in the future from boot time, versus what it was set to: > > >>>> > > >>>>>Firstboot-qemu-ga                        9/21/2023 4:04:00 PM   Ready > > >>>> > > >>>>Could a date or time field have not been swapped or been corrupted > > >>>>in some predictable way? > > >>> > > >>>Or in even simpler terms, what is the time (and timezone) that > > >>>this ^^^ machine was booted? > > >> > > >>I believe I have it figured out. > > >>The guest local time is currently 7:08 AM (a few minutes after > > >>firstboot/provisioning), pacific daylight time (UTC-7, though Windows > > >>displays it as "UTC-08:00"). This is the timezone that the guest comes > > >>configured with at first boot. The task is scheduled for 2:01 PM, > > >>meaning it's scheduled to run ~7 hours in the future. > > >> > > >>So it seems like the task was meant to be scheduled for 2:01 PM UTC (= > > >>7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*. > > >> > > >> From what I can see, the host machine time zone is irrelevant (UTC+2). > > >> > > >>I don't know where the timezone mixup comes from, though. Running > > >>`(get-date)` in the powershell at this point correctly returns the local > > >>time (7:08 AM). I guess during injection the time is in UTC, and > > >>schtasks.exe has no awareness of timezones? > > > > > >Right, I think there is a timezone disagreement between how we format the timestamp and how schtasks.exe takes it. > > > > > >What matters here is the /ST (start time) flag. > > > > > >Today we have (in the common submodule): > > > > > > add "$d = (get-date).AddSeconds(120)"; > > > add "$dtfinfo = [System.Globalization.DateTimeFormatInfo]::CurrentInfo"; > > > add "$sdp = $dtfinfo.ShortDatePattern"; > > > add "$sdp = $sdp -replace 'y+', 'yyyy'"; > > > add "$sdp = $sdp -replace 'M+', 'MM'"; > > > add "$sdp = $sdp -replace 'd+', 'dd'"; > > > add "schtasks.exe /Create /SC ONCE `"; > > > add " /ST $d.ToString('HH:mm') /SD $d.ToString($sdp) `"; > > > add " /RU SYSTEM /TN Firstboot-qemu-ga `"; > > > add (sprintf " /TR \"C:\\%s /forcerestart /qn /l+*vx C:\\%s.log\"" > > > msi_path msi_path); > > > > > >Note that for the /ST option's argument, we only perform the following steps: > > > > > > $d = (get-date).AddSeconds(120) > > > > > > /ST $d.ToString('HH:mm') > > > > > >This actually goes back to commit dc66e78fa37d ("windows: delay installation of qemu-ga MSI", 2020-03-10). The timestamp massaging we've since done only targeted the /SD (start date) option, not the start time (/ST) one! > > > > > >So the problem may be that > > > > > > (get-date).AddSeconds(120).ToString('HH:mm') > > > > > >formats the hour:minute timestamp in UTC (why though?), but the /ST option takes hour:minute in local time. > > > > > >Interestingly, DateTime objects seem to have a "Kind" property, which may be Utc, Local, or Unspec. > > > > > >https://learn.microsoft.com/en-us/dotnet/api/system.datetime.kind > > > > > >It seems to be used when converting from UTC to local or vice versa, and it probably influences how ToString() behaves too. I thought "get-date" returned a Local one, and /ST took a local one as well, but perhaps "get-date" returns a UTC timestamp in this case (when run from the script)? > > > > I think I have an idea what's happening. This is the part of the XML > > description of the guest: > > > > <clock offset="utc"/> > > Dan: > > For virt-v2v of Windows guests do you think we should always force > <clock offset="localtime"/> (but leave it at "utc" for non-Windows)? Generally 'localtime' is best for a default Windows install. IIUC, there is a way to tell Windows to assume the RTC is always in UTC, but its default behaviour is to assume localtime.

> I'm not clear what exactly should be the source of truth here. I > don't see anything in osinfo-db, unless I'm missing something. Yeah, we don't record anything in this area. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Hi Richard,        Sorry, I missed the email, I found out that the ova OS used by the customer is win11 uefi, so I tried to reproduce the error today using win11-uefi guest Steps: 1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with UTC-8 timezone from VMware by v2v, then check the installation status after finishing conversion Summary the test result as below:                                             qemu-ga   Win10-efi-UTC-8              PASS   Win11-non-efi UTC-8       PASS   Win11-efi-UTC-8              FAIL(can't find qemu-ga log in c:\)   Win11-efi-UTC+8             FAIL(can't find qemu-ga log in c:\)   Win2022-efi-UTC-8          PASS

  Based on the above results, I think this problem is only caused by win11-efi and has nothing to do with the time zone "UTC-8" The following are existing bugs about qemu-ga: Bug 1820144 - cannot install qemu-ga to some guests even if scheduled qemu-ga installation task exist Bug 2114809 - Can't install qemu-ga because of network address error after converting MD-RAID1 win11 host by virt-p2v Bug 1820152 - Fail to execute installation-qemu-ga.msi for win2012r2 and win8.1-i386 guests Thanks & Regards Ming Xie On Fri, Sep 22, 2023 at 6:40 PM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: > On 21.09.23 19:43, Richard W.M. Jones wrote: > >So this is probably another instance or variation of the timezone > >formatting problem (of schtasks).  Which version of virt-v2v is this? > >I want to check that you have a version with all the latest patches in > >this area. > > It's 2.2.0-1 from Debian (12) bookworm. I've verified that it > doesn't have any distro-specific patches. > > (https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/ debian > would have a patches/series file in this case) The timezone fixes are: commit 597d177567234c3a539098c423649781424eeb6f Author: Laszlo Ersek <lersek(a)redhat.com&gt; Date:   Tue Mar 8 15:30:51 2022 +0100     convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f Author: Richard W.M. Jones <rjones(a)redhat.com&gt; Date:   Fri Nov 12 08:47:55 2021 +0000     convert/convert_windows.ml: Handle date formats with dots instead of / They are all included in >= 2.0 I wonder if 597d177567 has a subtle flaw, or if we introduced a bug somewhere when refactoring this code later. Lee: Do you have a theory about exactly what is wrong with the schtasks date?  Like what was it supposed to be, assuming it was 120 seconds in the future from boot time, versus what it was set to: > Firstboot-qemu-ga                        9/21/2023 4:04:00 PM   Ready Could a date or time field have not been swapped or been corrupted in some predictable way? The code we run is here: https://github.com/libguestfs/libguestfs-common/blob/ e70d89a58dae068be2e19c7c21558707261af96a/mlcustomize/inject_virtio_win.ml# L571 Ming: this could be a bug affecting PST (UTC-8) guests, perhaps somehow related to having a single digit month field? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/ ~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines.  Supports shell scripting, bindings from many languages.  http://libguestfs.org

   Yes, I reproduced the problem using rhel9 virt-v2v, because I saw that the customer was also using rhel9 virt-v2v (version: 2.2), I saw that you wanted to solve the problem by fixing the  date of the schtask command, but I see that the schtask command used to install qemu-ga has been changed to the one shown in "qemu-ga-command-rhel9.png" on rhel9

  Besides,  qemu-ga can be installed successfully after converting win11-efi -UTC-8 guest on rhel8.9, which means I cannot reproduce the issue on rhel8 and the schtask command used for installing qemu-ga is old one as '' qemu-ga-command-rhel8.png' ' shown on rhel8.9, so I'm not sure if the fix can be work with rhel9 virt-v2v   Regards Ming Xie

Ming, For some reason the build seems "stuck" at the moment, but if it finishes could you see if this fixes the RHEL 9 case? https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547

On Thu, Sep 28, 2023 at 07:41:55PM +0800, Ming Xie wrote: > Hi Richard, > > On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: > > Ming, > > For some reason the build seems "stuck" at the moment, but > if it finishes could you see if this fixes the RHEL 9 case? > > https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547 > > > Sure, no problem, but I don't see the package available now, I will check later. The build seems stuck. I asked on the RHEL mailing list if someone can look at it. > BTW, is there a bug for this issue? Not at the moment.

On Thu, Sep 28, 2023 at 07:41:55PM +0800, Ming Xie wrote: > Hi Richard, > > On Thu, Sep 28, 2023 at 6:14 PM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: > > Ming, > > For some reason the build seems "stuck" at the moment, but > if it finishes could you see if this fixes the RHEL 9 case? > > https://kojihub.stream.centos.org/koji/taskinfo?taskID=2879547 It finished building now.

Hi Richard, Sorry, I missed the email, I found out that the ova OS used by the customer is win11 uefi, so I tried to reproduce the error today using win11-uefi guest Steps: 1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with UTC-8 timezone from VMware by v2v, then check the installation status after finishing conversion Summary the test result as below:                                             qemu-ga   Win10-efi-UTC-8              PASS   Win11-non-efi UTC-8       PASS   Win11-efi-UTC-8              FAIL(can't find qemu-ga log in c:\)   Win11-efi-UTC+8             FAIL(can't find qemu-ga log in c:\)   Win2022-efi-UTC-8          PASS   Based on the above results, I think this problem is only caused by win11-efi and has nothing to do with the time zone "UTC-8"

The following are existing bugs about qemu-ga: *Bug 1820144* <https://bugzilla.redhat.com/show_bug.cgi?id=1820144> - cannot install qemu-ga to some guests even if scheduled qemu-ga installation task exist *Bug 2114809* <https://bugzilla.redhat.com/show_bug.cgi?id=2114809> - Can't install qemu-ga because of network address error after converting MD-RAID1 win11 host by virt-p2v *Bug 1820152* <https://bugzilla.redhat.com/show_bug.cgi?id=1820152> - Fail to execute installation-qemu-ga.msi for win2012r2 and win8.1-i386 guests Thanks & Regards Ming Xie

(resend, as it bounced on tyan-all(a)redhat.com) Hi Ming, On 27.09.23 14:46, Ming Xie wrote: > Hi Richard, > > Sorry, I missed the email, I found out that the ova OS used by the customer is > win11 uefi, so I tried to reproduce the error today using win11-uefi guest > > Steps: > 1. Convert win10-efi with UTC-8 timezone, win11-non-efi with UTC-8 timezone, > win11-efi with UTC-8 timezone, win11-efi with UTC+8 timezone, win2022-efi with > UTC-8 timezone from VMware by v2v, then check the installation status after > finishing conversion > > > Summary the test result as below: > > qemu-ga > > Win10-efi-UTC-8 PASS > Win11-non-efi UTC-8 PASS > Win11-efi-UTC-8 FAIL(can't find qemu-ga log in c:\) > Win11-efi-UTC+8 FAIL(can't find qemu-ga log in c:\) > Win2022-efi-UTC-8 PASS > > Based on the above results, I think this problem is only caused by win11-efi > and has nothing to do with the time zone "UTC-8" hmm, where are you exactly setting the timezone? Within the guest, the scheduled task, or in the libvirt XML description?

The Windows 11 trial VM in question (downloaded from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ -> https://aka.ms/windev_VM_vmware) comes with the timezone within the VM set to US/Pacific. If I also set the RTC clock for the VM in the libvirt XML description after conversion like this: <clock offset="timezone" timezone="US/Pacific"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock> Then qemu-ga successfully gets installed after a 2 minute delay. If I leave the default libvirt XML description at: <clock offset="utc"/>