Possible issue with virt-v2v writing to wrong Windows registry control set
I may have stumbled across a virt-v2v bug. I just did a successful virt-v2v migration of a Windows 2008 server from an ESXi host to a RHEV environment. While the VM was still on the ESXi side, I removed its VMWare tools and rebooted it just make sure it was still functional. Then I shut it down and V2V'd and imported it into RHEV. When I first powered up the new RHEV copy, it found and installed the driver for the Virtio NIC and tried to install the Viostor SCSI driver, but failed with "Access Denied". This was strange. Looking at the Windows Device Manager, I noticed a SCSI controller without any driver and an "Unknown Device". This looked similar to the situation when it was an ESXi VM. I tinkered around with drivers a little bit, trying to be the Viostor stuff to load. It never did. So I rebooted the VM, just to make sure it would come back up while I was there to watch it and I could deal with its missing 2nd disk drive later. Good call - it went into a crash/reboot loop. Bummer. I caught it and tried "Last Known Good" from the Windows boot menu. This never works, but I keep trying anyway. Good thing, because this time it came right back up. It also successfully loaded the Viostor block driver this time and now there are no more SCSI controllers without drivers and no more Unidentified Devices. And that VM's second hard drive also showed up - this was probably the "unknown device" from before. I wonder if virt-v2v wrote its registry updates to the wrong registry control set and I stumbled across it? And a tip for anyone else - if your Windows VM crashes after a virt-v2v, try booting with Last Known Good. - Greg Scott
On 16/11/11 16:08, Greg Scott wrote:
I may have stumbled across a virt-v2v bug.
I just did a successful virt-v2v migration of a Windows 2008 server from an ESXi host to a RHEV environment. While the VM was still on the ESXi side, I removed its VMWare tools and rebooted it just make sure it was still functional. Then I shut it down and V2V’d and imported it into RHEV.
When I first powered up the new RHEV copy, it found and installed the driver for the Virtio NIC and tried to install the Viostor SCSI driver, but failed with “Access Denied”. This was strange. Looking at the Windows Device Manager, I noticed a SCSI controller without any driver and an “Unknown Device”. This looked similar to the situation when it was an ESXi VM.
I tinkered around with drivers a little bit, trying to be the Viostor stuff to load. It never did. So I rebooted the VM, just to make sure it would come back up while I was there to watch it and I could deal with its missing 2^nd disk drive later. Good call – it went into a crash/reboot loop. Bummer.
I caught it and tried “Last Known Good” from the Windows boot menu. This never works, but I keep trying anyway. Good thing, because this time it came right back up. It also successfully loaded the Viostor block driver this time and now there are no more SCSI controllers without drivers and no more Unidentified Devices. And that VM’s second hard drive also showed up – this was probably the “unknown device” from before.
I wonder if virt-v2v wrote its registry updates to the wrong registry control set and I stumbled across it?
And a tip for anyone else – if your Windows VM crashes after a virt-v2v, try booting with Last Known Good.
Can you reproduce this? Windows shouldn't be able to boot at all unless viostor is both installed and configured in the CDD of the correct control set. That is, afaict, the only way 'Last Known Good' would be bootable is if virt-v2v made it that way. That isn't to say there isn't something odd going on here... Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490
Can you reproduce this? Windows shouldn't be able to boot at all unless viostor is both installed and configured in the CDD of the correct control set. That is, afaict, the only way 'Last Known Good' would be bootable is if virt-v2v made it that way.
Reproducing the same conditions will be very difficult. The V2V'd VM is a member server in a Windows domain. It's now up and running and in production in the RHEV environment. But I can probably get permission and get my hands on the raw materials to set it up in a lab. This won't be easy; To reproduce it, we'd need an ESXi host, a RHEV environment complete with shared storage, and we'd need to copy the ESXi .VMDK files and provision a 40+ GB VM on the ESXi host and fire up a new V2V. It can be done but the logistics are very challenging and time consuming. I still have a copy of the V2V'd VM sitting in the export domain and I might be able to send over its virtual disks. I also have the ESXi .VMDK files I started from. This one came at the tail-end of a scenario that stumped lots of people - it's all documented in support case number 00560510. While still reasonably fresh in my head - here's the sequence of events to V2V this VM: 1 - The original ESXi VM had a bunch of snapshots and these need to collapse. So in the VMWare vSphere client, edit this VM and find the names of its .VMDK files. Now go to storage...browse datastore. Note that the original VM shows a bunch of .VMDK files because of the snapshots from a long time ago. Create a new directory and then copy and paste the old .VMDK files into this new directory. This creates 2 complete .VMDK files (Windows C and D drives in this VM), collapsing all the snapshots into these 2 new files. 2 - Provision a new ESXi VM using the new .VMDK files in the new directory - essentially a copy of the original ESXi VM but without snapshots. 3 - Boot the new VM, log on, and remove VMware tools from it. Shut the VM down. 4 - V2V it into the RHEV Export domain. 5 - Import it into the RHEV datacenter. I still have copies of all the various files left over along the way. But the ESXi stuff will disappear very soon because I have to repurpose that old ESXi host as a RHEV host tomorrow. - Greg
Windows shouldn't be able to boot at all unless viostor is both installed and configured in the CDD of the correct control set.
Yes, this was very strange. Looking at the Windows device manager when I first booted, I saw one viostor system drive and then a SCSI controller without a driver and the unknown device. Trying to install the viostor driver for that SCSI controller failed with "Access Denied" no matter what I tried. But the VM was still bootable, at least for a while, because it saw that viostor boot drive. How it saw the Viostor boot drive but not the 2nd drive is a mystery to me. But - doesn't the V2V process hack in some minimal viostor stuff to the new image file and then count on Windows properly loading up all the drivers after it boots? And there's an invisible firstboot at the tail end of the V2V to handle all this, right? If so, what if that minimal viostor hack was in the correct control set, but then all the complete driver stuff somehow ended up in the wrong control set? - Greg
On Thu, Nov 17, 2011 at 09:19:59AM -0600, Greg Scott wrote:
But - doesn't the V2V process hack in some minimal viostor stuff to the new image file and then count on Windows properly loading up all the drivers after it boots?
Yes.
And there's an invisible firstboot at the tail end of the V2V to handle all this, right? If so, what if that minimal viostor hack was in the correct control set, but then all the complete driver stuff somehow ended up in the wrong control set?
Fairly sure this shouldn't happen, or if it did it'd be a gross bug in the viostor installer. You can use virt-inspector to find the current control set for a Windows guest (requires libguestfs >= 1.10). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
I went back to the VMware copy of that converted VM - so this was the copy before virt-v2v. Opening up regedit and looking at HKLM\SYSTEM\Select - the value for keys Current and Default is 1. Failed is 0, LastKnownGood is 2. And then I was looking through /usr/share/perl5/vendor_perl/Sys/VirtV2V/Converter/Windows.pm. There's some code to figure out the CurrentControlSet: . . . my $h = Win::Hivex->open ($tmpdir . "/system", write => 1) or die "open system hive: $!"; # Get the 'Current' ControlSet. This is normally 001, but not always. my $select = $h->node_get_child($h->root(), 'Select'); my $current_cs = $h->node_get_value($select, 'Current'); $current_cs = sprintf("ControlSet%03i", $h->value_dword($current_cs)); . . . But then a few screens down, there's some code with ControlSet001 hard-coded: # Open the registry hive. my $h = Win::Hivex->open ($tmpdir . "/system", write => 1) or die "open system hive: $!"; # Make the changes. my $regedits = ' [HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\rhev-apt] "Type"=dword:00000010 "Start"=dword:00000002 "ErrorControl"=dword:00000001 "ImagePath"="c:\\\\Temp\\\\V2V\\\\rhsrvany.exe" "DisplayName"="RHSrvAny" "ObjectName"="LocalSystem" [HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\rhev-apt\Parameters] "CommandLine"="cmd /c \"c:\\\\Temp\\\\V2V\\\\firstboot.bat\"" "PWD"="c:\\\\Temp\\\\V2V" '; But none of this should have mattered on the VM I converted because the source VM CurrentControlSet apparently was ControlSet001. So hard-coded or calculated should not have mattered. Now looking at the production system after virt-v2v and after importing into RHEV - HKLM\SYSTEM\Select shows: Current and Default are 2, Failed is 1, LastKnownGood is 3. But this was after I booted with Last Known Good and it came up successfully. I still have a copy of the VM virtual disks before importing into RHEV sitting in the RHEV Export domain. Maybe there's a way to copy that image and provision a VM around it and look up its registry settings. - Greg
On Fri, Nov 18, 2011 at 08:21:32AM -0600, Greg Scott wrote:
But then a few screens down, there's some code with ControlSet001 hard-coded: [...] # Make the changes. my $regedits = ' [HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\rhev-apt]
This bug is fixed in the latest upstream version. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
participants (3)
-
Greg Scott -
Matthew Booth -
Richard W.M. Jones