10:59 p.m.
Now that we can gather diagnostic info, I think I know why our P2Vs kept
failing last week. Another one just died right in front of my eyes. I
think either the Ethernet or NFS server at this site occasionally
"blips" offline when it gets busy and that messes up P2V migrations.
The RHEV export domain is an NFS share offered by an old Storagetek NAS,
connected over a 10/100 Ethernet. All of us would prefer a gb Ethernet
but this is what we have available.
First, an extract from /var/log/messages showing the start and end times
for virt-p2v-server, about 3 1/2 hours.
[root@Fedora16-64P2V log]# tail /var/log/messages -c 1000
16-64P2V systemd-logind[704]: Removed session 16.
Nov 22 19:04:46 Fedora16-64P2V systemd-logind[704]: New session 17 of
user root.
Nov 22 19:04:48 Fedora16-64P2V virt-v2v[1733]: p2v-server started.
Nov 22 20:01:01 Fedora16-64P2V systemd-logind[704]: New session 18 of
user root.
Nov 22 20:01:02 Fedora16-64P2V systemd-logind[704]: Removed session 18.
Nov 22 21:01:01 Fedora16-64P2V systemd-logind[704]: New session 19 of
user root.
Nov 22 21:01:01 Fedora16-64P2V systemd-logind[704]: Removed session 19.
Nov 22 22:01:02 Fedora16-64P2V systemd-logind[704]: New session 20 of
user root.
Nov 22 22:01:02 Fedora16-64P2V systemd-logind[704]: Removed session 20.
Nov 22 22:33:44 Fedora16-64P2V virt-v2v[1733]: FATAL: Error receiving
data:
Nov 22 22:33:44 Fedora16-64P2V systemd-logind[704]: Removed session 17.
Nov 22 22:33:44 Fedora16-64P2V virt-v2v[1733]: WARNING: Error messages
were written to /var/log/virt-p2v-server.1322010288.log.
Nov 22 22:33:44 Fedora16-64P2V virt-v2v[1733]: p2v-server exited.
[root@Fedora16-64P2V log]#
Next, the output from the trace log file from virt-p2v-server:
[root@Fedora16-64P2V log]# more virt-p2v-server.1322010288.log
virt-v2v: Error receiving data:
[root@Fedora16-64P2V log]#
Before it blew up, I was watching network usage of my RHEV host with the
Fedora migration server VM and it consistently pegged above 90%+. I've
never seen an Ethernet that saturated before. The iftop utility on my
Fedora guest migration server showed data consistently flying at 69-90+
Mb per second.
- Greg Scott
10:21 p.m.
Trying that same P2V again, this time with everything connected to a
brand new HP Procurve 2910al-48G. All NICs on my hosts now show up at
1000 Gb now and this P2V still failed, this time after running 5 hours.
The source server is an aging HP Proliant ML110 running Windows 2003 (I
think). The conversion server is a 64 bit Fedora VM running in a RHEV
environment. The RHEV Export domain lives in an older Storagetek NAS.
For the next troubleshooting step, we'll brew up another NFS server from
somewhere and eliminate the Storagetek.
Here is an extract from /var/log/messages.
[root@Fedora16-64P2V log]# grep p2v-server messages
.
.
(Older unrelated messages removed)
.
.
Nov 23 14:55:59 Fedora16-64P2V virt-v2v[3452]: p2v-server started.
Nov 23 20:04:29 Fedora16-64P2V virt-v2v[3452]: WARNING: Error messages
were written to /var/log/virt-p2v-server.1322081759.log.
Nov 23 20:04:29 Fedora16-64P2V virt-v2v[3452]: p2v-server exited.
[root@Fedora16-64P2V log]#
And here is the trace log file from virt-p2v-server:
[root@Fedora16-64P2V log]# more virt-p2v-server.1322081759.log
virt-v2v: Error receiving data:
[root@Fedora16-64P2V log]#
- Greg
5:45 a.m.
We've had one successful P2V with a test XP workstation and several
failures with a Windows 2003 Server. Both P2Vs use an old Storagetek
NAS for NFS storage for the RHEV Export domain. Both use my bleeding
edge 64 bit Fedora 16 RHEV VM as a migration server. Both boot from the
exact same same P2V CD that we built a few days ago.
So what's different between the successful source XP workstation and the
failing source 2003 server?
The failing source 2003 server is an HP Proliant ML110 with 2 hard
drives. I wonder what that ML110 uses for disk controller? Probably one
of those low end HP fakeRAID controllers - or they might just be plain
old fashioned SATA drives. The successful source XP workstation only
has one hard drive. Are there any known P2V issues migrating source
machines with multiple hard drives? What if the source server copied
the first hard drive but then blew up trying to start the 2nd hard
drive? See below for the only footprints I can find for any diagnostic
data.
This one is maddening because it runs for several hours and then decides
to go belly up, leaving very little diagnostic info behind.
We've been chasing network issues - but maybe the network and
infrastruture is fine and something is going on with the source server.
How do we check this?
- Greg
-----Original Message-----
From: libguestfs-bounces(a)redhat.com
[mailto:libguestfs-bounces@redhat.com] On Behalf Of Greg Scott
Sent: Wednesday, November 23, 2011 10:21 PM
To: libguestfs(a)redhat.com
Cc: Fredy Hernandez; Joey Bertalan
Subject: Re: [Libguestfs] P2Vs seem to require a very robust Ethernet
Trying that same P2V again, this time with everything connected to a
brand new HP Procurve 2910al-48G. All NICs on my hosts now show up at
1000 Gb now and this P2V still failed, this time after running 5 hours.
The source server is an aging HP Proliant ML110 running Windows 2003 (I
think). The conversion server is a 64 bit Fedora VM running in a RHEV
environment. The RHEV Export domain lives in an older Storagetek NAS.
For the next troubleshooting step, we'll brew up another NFS server from
somewhere and eliminate the Storagetek.
Here is an extract from /var/log/messages.
[root@Fedora16-64P2V log]# grep p2v-server messages
.
.
(Older unrelated messages removed)
.
.
Nov 23 14:55:59 Fedora16-64P2V virt-v2v[3452]: p2v-server started.
Nov 23 20:04:29 Fedora16-64P2V virt-v2v[3452]: WARNING: Error messages
were written to /var/log/virt-p2v-server.1322081759.log.
Nov 23 20:04:29 Fedora16-64P2V virt-v2v[3452]: p2v-server exited.
[root@Fedora16-64P2V log]#
And here is the trace log file from virt-p2v-server:
[root@Fedora16-64P2V log]# more virt-p2v-server.1322081759.log
virt-v2v: Error receiving data:
[root@Fedora16-64P2V log]#
- Greg
_______________________________________________
Libguestfs mailing list
Libguestfs(a)redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
5:51 a.m.
On 11/24/2011 11:45 AM, Greg Scott wrote:
We've had one successful P2V with a test XP workstation and
several
failures with a Windows 2003 Server. Both P2Vs use an old Storagetek
NAS for NFS storage for the RHEV Export domain. Both use my bleeding
edge 64 bit Fedora 16 RHEV VM as a migration server. Both boot from the
exact same same P2V CD that we built a few days ago.
So what's different between the successful source XP workstation and the
failing source 2003 server?
The failing source 2003 server is an HP Proliant ML110 with 2 hard
drives. I wonder what that ML110 uses for disk controller? Probably one
of those low end HP fakeRAID controllers - or they might just be plain
old fashioned SATA drives. The successful source XP workstation only
has one hard drive. Are there any known P2V issues migrating source
machines with multiple hard drives? What if the source server copied
the first hard drive but then blew up trying to start the 2nd hard
drive? See below for the only footprints I can find for any diagnostic
data.
This one is maddening because it runs for several hours and then decides
to go belly up, leaving very little diagnostic info behind.
We've been chasing network issues - but maybe the network and
infrastruture is fine and something is going on with the source server.
How do we check this?
I'm currently working on getting a new release out. I can't say it will
fix this problem, but the new P2V client might help get better
diagnostic info from the source server.
Hopefully within the next day or two.
Matt
--
Matthew Booth, RHCA, RHCSS
Red Hat Engineering, Virtualisation Team
GPG ID: D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490
10:51 a.m.
.
.
.
> We've been chasing network issues - but maybe the network
and
> infrastruture is fine and something is going on with the source
server.
> How do we check this?
I'm currently working on getting a new release out. I can't
say it
will
fix this problem, but the new P2V client might help get better
diagnostic info from the source server.
Should I be looking for a new virt-p2v-image-builder and then burn
another CD?
Thanks
- Greg
11:10 a.m.
On 11/28/2011 04:51 PM, Greg Scott wrote:
.
.
.
>> We've been chasing network issues - but maybe the network and
>> infrastruture is fine and something is going on with the source
server.
>> How do we check this?
> I'm currently working on getting a new release out. I can't say it
will
> fix this problem, but the new P2V client might help get better
> diagnostic info from the source server.
Should I be looking for a new virt-p2v-image-builder and then burn
another CD?
Yes. Unfortunately we've got a bit of a dependency tree going on, so
it'll take a few days at least to get a build out. I'll update the list
when I do.
Matt
--
Matthew Booth, RHCA, RHCSS
Red Hat Engineering, Virtualisation Team
GPG ID: D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490
4:54 a.m.
OK, thanks.
For what it's worth, we just had another P2V attempt die. This source
system is an old HP Pavillion with Windows Server 2003 installed. The
P2V hung on the source server at 13 percent. Looking at the log file on
the Fedora conversion server:
[root@Fedora16-64P2V log]# more virt-p2v-server.1322503432.log
virt-v2v: Error receiving data:
[root@Fedora16-64P2V log]# ps ax | grep p2v
9375 pts/0 S+ 0:00 grep --color=auto p2v
[root@Fedora16-64P2V log]#
Hopefully the email didn't garble the text above. As you can see,
virt-p2v-server on the conversion server died somewhere along the way.
- Greg
9:12 p.m.
Hey Matt -
With a new release of virt-v2v a while away, what if we were to try a
P2V by hand? I'm thinking of maybe provisioning a Windows RHEV VM,
doing an ntbackup of the source physical machine and doing a bare metal
restore on the VM. I've done it that way before with mixed results.
We have that hack I put together to load the viostor block driver stuff
on the source machine so the target VM will boot. Are there any HAL
kinds of things or other hardware/firmware issues we need to also be
thinking about?
Thanks
- Greg
3:28 a.m.
On 12/02/2011 03:12 AM, Greg Scott wrote:
Hey Matt -
With a new release of virt-v2v a while away, what if we were to try a
P2V by hand? I'm thinking of maybe provisioning a Windows RHEV VM,
doing an ntbackup of the source physical machine and doing a bare metal
restore on the VM. I've done it that way before with mixed results.
We have that hack I put together to load the viostor block driver stuff
on the source machine so the target VM will boot. Are there any HAL
kinds of things or other hardware/firmware issues we need to also be
thinking about?
Windows mostly fixes itself. If it boots and you don't see any missing
drivers in Device Manager, you're probably good to go.
Matt
--
Matthew Booth, RHCA, RHCSS
Red Hat Engineering, Virtualisation Team
GPG ID: D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490
4771
days inactive
4780
days old
8 comments
2 participants
participants (2)
-
Greg Scott -
Matthew Booth