10:08 a.m.
I may have stumbled across a virt-v2v bug.
I just did a successful virt-v2v migration of a Windows 2008 server from
an ESXi host to a RHEV environment. While the VM was still on the ESXi
side, I removed its VMWare tools and rebooted it just make sure it was
still functional. Then I shut it down and V2V'd and imported it into
RHEV.
When I first powered up the new RHEV copy, it found and installed the
driver for the Virtio NIC and tried to install the Viostor SCSI driver,
but failed with "Access Denied". This was strange. Looking at the
Windows Device Manager, I noticed a SCSI controller without any driver
and an "Unknown Device". This looked similar to the situation when it
was an ESXi VM.
I tinkered around with drivers a little bit, trying to be the Viostor
stuff to load. It never did. So I rebooted the VM, just to make sure
it would come back up while I was there to watch it and I could deal
with its missing 2nd disk drive later. Good call - it went into a
crash/reboot loop. Bummer.
I caught it and tried "Last Known Good" from the Windows boot menu.
This never works, but I keep trying anyway. Good thing, because this
time it came right back up. It also successfully loaded the Viostor
block driver this time and now there are no more SCSI controllers
without drivers and no more Unidentified Devices. And that VM's second
hard drive also showed up - this was probably the "unknown device" from
before.
I wonder if virt-v2v wrote its registry updates to the wrong registry
control set and I stumbled across it?
And a tip for anyone else - if your Windows VM crashes after a virt-v2v,
try booting with Last Known Good.
- Greg Scott
3:20 a.m.
On 16/11/11 16:08, Greg Scott wrote:
I may have stumbled across a virt-v2v bug.
I just did a successful virt-v2v migration of a Windows 2008 server from
an ESXi host to a RHEV environment. While the VM was still on the ESXi
side, I removed its VMWare tools and rebooted it just make sure it was
still functional. Then I shut it down and V2V’d and imported it into RHEV.
When I first powered up the new RHEV copy, it found and installed the
driver for the Virtio NIC and tried to install the Viostor SCSI driver,
but failed with “Access Denied”. This was strange. Looking at the
Windows Device Manager, I noticed a SCSI controller without any driver
and an “Unknown Device”. This looked similar to the situation when it
was an ESXi VM.
I tinkered around with drivers a little bit, trying to be the Viostor
stuff to load. It never did. So I rebooted the VM, just to make sure it
would come back up while I was there to watch it and I could deal with
its missing 2^nd disk drive later. Good call – it went into a
crash/reboot loop. Bummer.
I caught it and tried “Last Known Good” from the Windows boot menu. This
never works, but I keep trying anyway. Good thing, because this time it
came right back up. It also successfully loaded the Viostor block driver
this time and now there are no more SCSI controllers without drivers and
no more Unidentified Devices. And that VM’s second hard drive also
showed up – this was probably the “unknown device” from before.
I wonder if virt-v2v wrote its registry updates to the wrong registry
control set and I stumbled across it?
And a tip for anyone else – if your Windows VM crashes after a virt-v2v,
try booting with Last Known Good.
Can you reproduce this? Windows shouldn't be able to boot at all unless
viostor is both installed and configured in the CDD of the correct
control set. That is, afaict, the only way 'Last Known Good' would be
bootable is if virt-v2v made it that way.
That isn't to say there isn't something odd going on here...
Matt
--
Matthew Booth, RHCA, RHCSS
Red Hat Engineering, Virtualisation Team
GPG ID: D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490
9:08 a.m.
Can you reproduce this? Windows shouldn't be able to boot at all
unless
viostor is both installed and configured in the CDD of the correct
control set. That is, afaict, the only way 'Last Known Good' would be
bootable is if virt-v2v made it that way.
Reproducing the same conditions will be very difficult. The V2V'd VM is
a member server in a Windows domain. It's now up and running and in
production in the RHEV environment. But I can probably get permission
and get my hands on the raw materials to set it up in a lab. This won't
be easy; To reproduce it, we'd need an ESXi host, a RHEV environment
complete with shared storage, and we'd need to copy the ESXi .VMDK files
and provision a 40+ GB VM on the ESXi host and fire up a new V2V. It
can be done but the logistics are very challenging and time consuming.
I still have a copy of the V2V'd VM sitting in the export domain and I
might be able to send over its virtual disks. I also have the ESXi
.VMDK files I started from. This one came at the tail-end of a scenario
that stumped lots of people - it's all documented in support case number
00560510.
While still reasonably fresh in my head - here's the sequence of events
to V2V this VM:
1 - The original ESXi VM had a bunch of snapshots and these need to
collapse. So in the VMWare vSphere client, edit this VM and find the
names of its .VMDK files. Now go to storage...browse datastore. Note
that the original VM shows a bunch of .VMDK files because of the
snapshots from a long time ago. Create a new directory and then copy
and paste the old .VMDK files into this new directory. This creates 2
complete .VMDK files (Windows C and D drives in this VM), collapsing all
the snapshots into these 2 new files.
2 - Provision a new ESXi VM using the new .VMDK files in the new
directory - essentially a copy of the original ESXi VM but without
snapshots.
3 - Boot the new VM, log on, and remove VMware tools from it. Shut the
VM down.
4 - V2V it into the RHEV Export domain.
5 - Import it into the RHEV datacenter.
I still have copies of all the various files left over along the way.
But the ESXi stuff will disappear very soon because I have to repurpose
that old ESXi host as a RHEV host tomorrow.
- Greg
9:19 a.m.
Windows shouldn't be able to boot at all unless viostor is both
installed and configured
in the CDD of the correct control set.
Yes, this was very strange. Looking at the Windows device manager when
I first booted, I saw one viostor system drive and then a SCSI
controller without a driver and the unknown device. Trying to install
the viostor driver for that SCSI controller failed with "Access Denied"
no matter what I tried. But the VM was still bootable, at least for a
while, because it saw that viostor boot drive. How it saw the Viostor
boot drive but not the 2nd drive is a mystery to me.
But - doesn't the V2V process hack in some minimal viostor stuff to the
new image file and then count on Windows properly loading up all the
drivers after it boots? And there's an invisible firstboot at the tail
end of the V2V to handle all this, right? If so, what if that minimal
viostor hack was in the correct control set, but then all the complete
driver stuff somehow ended up in the wrong control set?
- Greg
2:25 p.m.
On Thu, Nov 17, 2011 at 09:19:59AM -0600, Greg Scott wrote:
But - doesn't the V2V process hack in some minimal viostor stuff
to the
new image file and then count on Windows properly loading up all the
drivers after it boots?
Yes.
And there's an invisible firstboot at the tail end of the V2V to
handle all this, right? If so, what if that minimal viostor hack
was in the correct control set, but then all the complete driver
stuff somehow ended up in the wrong control set?
Fairly sure this shouldn't happen, or if it did it'd be a gross bug in
the viostor installer.
You can use virt-inspector to find the current control set for a
Windows guest (requires libguestfs >= 1.10).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v
8:21 a.m.
I went back to the VMware copy of that converted VM - so this was the
copy before virt-v2v. Opening up regedit and looking at
HKLM\SYSTEM\Select - the value for keys Current and Default is 1.
Failed is 0, LastKnownGood is 2.
And then I was looking through
/usr/share/perl5/vendor_perl/Sys/VirtV2V/Converter/Windows.pm.
There's some code to figure out the CurrentControlSet:
.
.
.
my $h = Win::Hivex->open ($tmpdir . "/system", write => 1)
or die "open system hive: $!";
# Get the 'Current' ControlSet. This is normally 001, but not
always.
my $select = $h->node_get_child($h->root(), 'Select');
my $current_cs = $h->node_get_value($select, 'Current');
$current_cs = sprintf("ControlSet%03i",
$h->value_dword($current_cs));
.
.
.
But then a few screens down, there's some code with ControlSet001
hard-coded:
# Open the registry hive.
my $h = Win::Hivex->open ($tmpdir . "/system", write => 1)
or die "open system hive: $!";
# Make the changes.
my $regedits = '
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\rhev-apt]
"Type"=dword:00000010
"Start"=dword:00000002
"ErrorControl"=dword:00000001
"ImagePath"="c:\\\\Temp\\\\V2V\\\\rhsrvany.exe"
"DisplayName"="RHSrvAny"
"ObjectName"="LocalSystem"
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\rhev-apt\Parameters]
"CommandLine"="cmd /c
\"c:\\\\Temp\\\\V2V\\\\firstboot.bat\""
"PWD"="c:\\\\Temp\\\\V2V"
';
But none of this should have mattered on the VM I converted because the
source VM CurrentControlSet apparently was ControlSet001. So hard-coded
or calculated should not have mattered.
Now looking at the production system after virt-v2v and after importing
into RHEV -
HKLM\SYSTEM\Select shows: Current and Default are 2, Failed is 1,
LastKnownGood is 3. But this was after I booted with Last Known Good
and it came up successfully.
I still have a copy of the VM virtual disks before importing into RHEV
sitting in the RHEV Export domain. Maybe there's a way to copy that
image and provision a VM around it and look up its registry settings.
- Greg
8:53 a.m.
On Fri, Nov 18, 2011 at 08:21:32AM -0600, Greg Scott wrote:
But then a few screens down, there's some code with
ControlSet001
hard-coded:
[...]
# Make the changes.
my $regedits = '
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\rhev-apt]
This bug is fixed in the latest upstream version.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/
4752
days inactive
4754
days old
6 comments
3 participants
participants (3)
-
Greg Scott -
Matthew Booth -
Richard W.M. Jones