P2Vs seem to require a very robust Ethernet
Now that we can gather diagnostic info, I think I know why our P2Vs kept failing last week. Another one just died right in front of my eyes. I think either the Ethernet or NFS server at this site occasionally "blips" offline when it gets busy and that messes up P2V migrations. The RHEV export domain is an NFS share offered by an old Storagetek NAS, connected over a 10/100 Ethernet. All of us would prefer a gb Ethernet but this is what we have available. First, an extract from /var/log/messages showing the start and end times for virt-p2v-server, about 3 1/2 hours. [root@Fedora16-64P2V log]# tail /var/log/messages -c 1000 16-64P2V systemd-logind[704]: Removed session 16. Nov 22 19:04:46 Fedora16-64P2V systemd-logind[704]: New session 17 of user root. Nov 22 19:04:48 Fedora16-64P2V virt-v2v[1733]: p2v-server started. Nov 22 20:01:01 Fedora16-64P2V systemd-logind[704]: New session 18 of user root. Nov 22 20:01:02 Fedora16-64P2V systemd-logind[704]: Removed session 18. Nov 22 21:01:01 Fedora16-64P2V systemd-logind[704]: New session 19 of user root. Nov 22 21:01:01 Fedora16-64P2V systemd-logind[704]: Removed session 19. Nov 22 22:01:02 Fedora16-64P2V systemd-logind[704]: New session 20 of user root. Nov 22 22:01:02 Fedora16-64P2V systemd-logind[704]: Removed session 20. Nov 22 22:33:44 Fedora16-64P2V virt-v2v[1733]: FATAL: Error receiving data: Nov 22 22:33:44 Fedora16-64P2V systemd-logind[704]: Removed session 17. Nov 22 22:33:44 Fedora16-64P2V virt-v2v[1733]: WARNING: Error messages were written to /var/log/virt-p2v-server.1322010288.log. Nov 22 22:33:44 Fedora16-64P2V virt-v2v[1733]: p2v-server exited. [root@Fedora16-64P2V log]# Next, the output from the trace log file from virt-p2v-server: [root@Fedora16-64P2V log]# more virt-p2v-server.1322010288.log virt-v2v: Error receiving data: [root@Fedora16-64P2V log]# Before it blew up, I was watching network usage of my RHEV host with the Fedora migration server VM and it consistently pegged above 90%+. I've never seen an Ethernet that saturated before. The iftop utility on my Fedora guest migration server showed data consistently flying at 69-90+ Mb per second. - Greg Scott
Trying that same P2V again, this time with everything connected to a brand new HP Procurve 2910al-48G. All NICs on my hosts now show up at 1000 Gb now and this P2V still failed, this time after running 5 hours. The source server is an aging HP Proliant ML110 running Windows 2003 (I think). The conversion server is a 64 bit Fedora VM running in a RHEV environment. The RHEV Export domain lives in an older Storagetek NAS. For the next troubleshooting step, we'll brew up another NFS server from somewhere and eliminate the Storagetek. Here is an extract from /var/log/messages. [root@Fedora16-64P2V log]# grep p2v-server messages . . (Older unrelated messages removed) . . Nov 23 14:55:59 Fedora16-64P2V virt-v2v[3452]: p2v-server started. Nov 23 20:04:29 Fedora16-64P2V virt-v2v[3452]: WARNING: Error messages were written to /var/log/virt-p2v-server.1322081759.log. Nov 23 20:04:29 Fedora16-64P2V virt-v2v[3452]: p2v-server exited. [root@Fedora16-64P2V log]# And here is the trace log file from virt-p2v-server: [root@Fedora16-64P2V log]# more virt-p2v-server.1322081759.log virt-v2v: Error receiving data: [root@Fedora16-64P2V log]# - Greg
We've had one successful P2V with a test XP workstation and several failures with a Windows 2003 Server. Both P2Vs use an old Storagetek NAS for NFS storage for the RHEV Export domain. Both use my bleeding edge 64 bit Fedora 16 RHEV VM as a migration server. Both boot from the exact same same P2V CD that we built a few days ago. So what's different between the successful source XP workstation and the failing source 2003 server? The failing source 2003 server is an HP Proliant ML110 with 2 hard drives. I wonder what that ML110 uses for disk controller? Probably one of those low end HP fakeRAID controllers - or they might just be plain old fashioned SATA drives. The successful source XP workstation only has one hard drive. Are there any known P2V issues migrating source machines with multiple hard drives? What if the source server copied the first hard drive but then blew up trying to start the 2nd hard drive? See below for the only footprints I can find for any diagnostic data. This one is maddening because it runs for several hours and then decides to go belly up, leaving very little diagnostic info behind. We've been chasing network issues - but maybe the network and infrastruture is fine and something is going on with the source server. How do we check this? - Greg -----Original Message----- From: libguestfs-bounces@redhat.com [mailto:libguestfs-bounces@redhat.com] On Behalf Of Greg Scott Sent: Wednesday, November 23, 2011 10:21 PM To: libguestfs@redhat.com Cc: Fredy Hernandez; Joey Bertalan Subject: Re: [Libguestfs] P2Vs seem to require a very robust Ethernet Trying that same P2V again, this time with everything connected to a brand new HP Procurve 2910al-48G. All NICs on my hosts now show up at 1000 Gb now and this P2V still failed, this time after running 5 hours. The source server is an aging HP Proliant ML110 running Windows 2003 (I think). The conversion server is a 64 bit Fedora VM running in a RHEV environment. The RHEV Export domain lives in an older Storagetek NAS. For the next troubleshooting step, we'll brew up another NFS server from somewhere and eliminate the Storagetek. Here is an extract from /var/log/messages. [root@Fedora16-64P2V log]# grep p2v-server messages . . (Older unrelated messages removed) . . Nov 23 14:55:59 Fedora16-64P2V virt-v2v[3452]: p2v-server started. Nov 23 20:04:29 Fedora16-64P2V virt-v2v[3452]: WARNING: Error messages were written to /var/log/virt-p2v-server.1322081759.log. Nov 23 20:04:29 Fedora16-64P2V virt-v2v[3452]: p2v-server exited. [root@Fedora16-64P2V log]# And here is the trace log file from virt-p2v-server: [root@Fedora16-64P2V log]# more virt-p2v-server.1322081759.log virt-v2v: Error receiving data: [root@Fedora16-64P2V log]# - Greg _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
On 11/24/2011 11:45 AM, Greg Scott wrote:
We've had one successful P2V with a test XP workstation and several failures with a Windows 2003 Server. Both P2Vs use an old Storagetek NAS for NFS storage for the RHEV Export domain. Both use my bleeding edge 64 bit Fedora 16 RHEV VM as a migration server. Both boot from the exact same same P2V CD that we built a few days ago.
So what's different between the successful source XP workstation and the failing source 2003 server?
The failing source 2003 server is an HP Proliant ML110 with 2 hard drives. I wonder what that ML110 uses for disk controller? Probably one of those low end HP fakeRAID controllers - or they might just be plain old fashioned SATA drives. The successful source XP workstation only has one hard drive. Are there any known P2V issues migrating source machines with multiple hard drives? What if the source server copied the first hard drive but then blew up trying to start the 2nd hard drive? See below for the only footprints I can find for any diagnostic data.
This one is maddening because it runs for several hours and then decides to go belly up, leaving very little diagnostic info behind.
We've been chasing network issues - but maybe the network and infrastruture is fine and something is going on with the source server. How do we check this?
I'm currently working on getting a new release out. I can't say it will fix this problem, but the new P2V client might help get better diagnostic info from the source server. Hopefully within the next day or two. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490
. . .
We've been chasing network issues - but maybe the network and infrastruture is fine and something is going on with the source server. How do we check this?
I'm currently working on getting a new release out. I can't say it will fix this problem, but the new P2V client might help get better diagnostic info from the source server.
Should I be looking for a new virt-p2v-image-builder and then burn another CD? Thanks - Greg
On 11/28/2011 04:51 PM, Greg Scott wrote:
. . .
We've been chasing network issues - but maybe the network and infrastruture is fine and something is going on with the source server. How do we check this?
I'm currently working on getting a new release out. I can't say it will fix this problem, but the new P2V client might help get better diagnostic info from the source server.
Should I be looking for a new virt-p2v-image-builder and then burn another CD?
Yes. Unfortunately we've got a bit of a dependency tree going on, so it'll take a few days at least to get a build out. I'll update the list when I do. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490
OK, thanks. For what it's worth, we just had another P2V attempt die. This source system is an old HP Pavillion with Windows Server 2003 installed. The P2V hung on the source server at 13 percent. Looking at the log file on the Fedora conversion server: [root@Fedora16-64P2V log]# more virt-p2v-server.1322503432.log virt-v2v: Error receiving data: [root@Fedora16-64P2V log]# ps ax | grep p2v 9375 pts/0 S+ 0:00 grep --color=auto p2v [root@Fedora16-64P2V log]# Hopefully the email didn't garble the text above. As you can see, virt-p2v-server on the conversion server died somewhere along the way. - Greg
Hey Matt - With a new release of virt-v2v a while away, what if we were to try a P2V by hand? I'm thinking of maybe provisioning a Windows RHEV VM, doing an ntbackup of the source physical machine and doing a bare metal restore on the VM. I've done it that way before with mixed results. We have that hack I put together to load the viostor block driver stuff on the source machine so the target VM will boot. Are there any HAL kinds of things or other hardware/firmware issues we need to also be thinking about? Thanks - Greg
On 12/02/2011 03:12 AM, Greg Scott wrote:
Hey Matt -
With a new release of virt-v2v a while away, what if we were to try a P2V by hand? I'm thinking of maybe provisioning a Windows RHEV VM, doing an ntbackup of the source physical machine and doing a bare metal restore on the VM. I've done it that way before with mixed results.
We have that hack I put together to load the viostor block driver stuff on the source machine so the target VM will boot. Are there any HAL kinds of things or other hardware/firmware issues we need to also be thinking about?
Windows mostly fixes itself. If it boots and you don't see any missing drivers in Device Manager, you're probably good to go. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490
participants (2)
-
Greg Scott -
Matthew Booth