Re: [Libguestfs] P2Vs seem to require a very robust Ethernet
Trying a P2V again, this time using a physical Fedora system for the NFS RHEV Export domain instead of the Storagetek NFS server. Everything else is the same. Same source server, same conversion server, same v2v and p2v versions for now. The only variable different is the RHEV Export NFS server.
This time, the P2V finished at 10:36 PM. Start time was 8:38 PM. Total size was 74 GB over roughly 2 hours of elapsed time. I just started a RHEV import and that should finish a couple of hours from now. - Greg
On Tue, Dec 06, 2011 at 01:02:49AM -0600, Greg Scott wrote:
Trying a P2V again, this time using a physical Fedora system for the NFS RHEV Export domain instead of the Storagetek NFS server. Everything else is the same. Same source server, same conversion server, same v2v and p2v versions for now. The only variable different is the RHEV Export NFS server.
This time, the P2V finished at 10:36 PM. Start time was 8:38 PM. Total size was 74 GB over roughly 2 hours of elapsed time. I just started a RHEV import and that should finish a couple of hours from now.
Greg, is this with Matt's latest F16 packages? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
Greg, is this with Matt's latest F16 packages?
No. Not yet. I only wanted to change one thing at a time so right this second I'm still using Matt's special 0.8.4.1 from a few days ago. But now I think I need to try again with 0.8.5 (I think that's the version.) Trying to import that VM into RHEV keeps failing. Every time I start an import, RHEV fails it about 10 minutes into the import. Then both my Export domain and ISO domain go offline. Curiously, the RHEV Export domain is now on that Fedora system and the ISO domain is still on the Storagetek. But they both go offline every time I start a RHEV import. This gets even better. I have 2 hosts named thing1 and thing2. Host thing1 was the SPM (Storage Pool Manager), but apparently went offline. Well, sort of. Host thing1 had several VMs and I successfully migrated all of them to host thing2. Then I rebooted thing1. Both hosts are HP Proliants and I can get at their consoles via an ILO. Connecting to thing1 via ILO, it can no longer ping anywhere. When thing2 tries to ping thing1, watching on thing1 with tcpdump, I see the echo requests come in on bridge device rhevm and on physical eth0. But thing1 never sends any replies. Listening in general with tcpdump on eth0, thing1 goes into PROMISC mode and I see traffic from all over the LAN. But thing1 never sends anything out. It listens but apparently won't talk. Well, OK, so maybe I have a some kind of hardware problem with thing1. Bad motherboard? But now host thing2 has all the VMs and the SPM role. I should be able to do a RHEV import using thing2 even if thing1 has a problem. But this consistently fails after about 10 minutes when RHEV-M logs a failure message and takes both the ISO and Export domains offline. Starting up a RHEV import and watching with tcpdump on thing2 - this is strange. Here is a sample. Thing2 is 175.10.0.62 and the Fedora NFS server is 175.10.0.95. I wonder what ERR 1448 means? . . . 09:51:05.213385 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53958549 win 24571 <nop,nop,timestamp 1083651691 57298290> 09:51:05.213501 IP 175.10.0.95.2049 > 175.10.0.62.1600941934: reply ERR 1448 09:51:05.213626 IP 175.10.0.95.2049 > 175.10.0.62.92276579: reply ERR 1448 09:51:05.213632 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53961445 win 24571 <nop,nop,timestamp 1083651692 57298290> 09:51:05.213748 IP 175.10.0.95.2049 > 175.10.0.62.1969553664: reply ERR 1448 09:51:05.213871 IP 175.10.0.95.2049 > 175.10.0.62.2675744636: reply ERR 1448 09:51:05.213878 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53964341 win 24571 <nop,nop,timestamp 1083651692 57298290> 09:51:05.213994 IP 175.10.0.95.2049 > 175.10.0.62.1366776504: reply ERR 1448 09:51:05.214117 IP 175.10.0.95.2049 > 175.10.0.62.1668246831: reply ERR 1448 09:51:05.214124 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53967237 win 24571 <nop,nop,timestamp 1083651692 57298290> 09:51:05.214240 IP 175.10.0.95.2049 > 175.10.0.62.1718183282: reply ERR 1448 09:51:05.214363 IP 175.10.0.95.2049 > 175.10.0.62.1997868874: reply ERR 1448 09:51:05.214374 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53970133 win 24571 <nop,nop,timestamp 1083651692 57298291> 09:51:05.214485 IP 175.10.0.95.2049 > 175.10.0.62.1903301888: reply ERR 1448 09:51:05.214610 IP 175.10.0.95.2049 > 175.10.0.62.1254097601: reply ERR 1448 09:51:05.214617 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53973029 win 24571 <nop,nop,timestamp 1083651693 57298291> 09:51:05.214733 IP 175.10.0.95.2049 > 175.10.0.62.16778812: reply ERR 1448 09:51:05.214856 IP 175.10.0.95.2049 > 175.10.0.62.524340: reply ERR 1448 09:51:05.214863 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53975925 win 24571 <nop,nop,timestamp 1083651693 57298291> 09:51:05.214978 IP 175.10.0.95.2049 > 175.10.0.62.2802450528: reply ERR 1448 09:51:05.215102 IP 175.10.0.95.2049 > 175.10.0.62.1952800512: reply ERR 1448 09:51:05.215108 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53978821 win 24571 <nop,nop,timestamp 1083651693 57298291> 09:51:05.215224 IP 175.10.0.95.2049 > 175.10.0.62.3376141: reply ERR 1448 09:51:05.215347 IP 175.10.0.95.2049 > 175.10.0.62.201329664: reply ERR 1448 09:51:05.215355 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53981717 win 24571 <nop,nop,timestamp 1083651693 57298291> 09:51:05.215470 IP 175.10.0.95.2049 > 175.10.0.62.1176568680: reply ERR 1448 09:51:05.215594 IP 175.10.0.95.2049 > 175.10.0.62.3541536: reply ERR 1448 09:51:05.215601 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53984613 win 24571 <nop,nop,timestamp 1083651694 57298292> 09:51:05.215717 IP 175.10.0.95.2049 > 175.10.0.62.3758274048: reply ERR 1448 09:51:05.215840 IP 175.10.0.95.2049 > 175.10.0.62.570522807: reply ERR 1448 09:51:05.215847 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53987509 win 24571 <nop,nop,timestamp 1083651694 57298292> 09:51:05.215962 IP 175.10.0.95.2049 > 175.10.0.62.2183504323: reply ERR 1448 09:51:05.216085 IP 175.10.0.95.2049 > 175.10.0.62.117456135: reply ERR 1448 09:51:05.216092 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53990405 win 24571 <nop,nop,timestamp 1083651694 57298292> 09:51:05.216209 IP 175.10.0.95.2049 > 175.10.0.62.41447938: reply ERR 1448 09:51:05.216332 IP 175.10.0.95.2049 > 175.10.0.62.74776836: reply ERR 1448 09:51:05.216341 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53993301 win 24571 <nop,nop,timestamp 1083651694 57298292> 09:51:05.216454 IP 175.10.0.95.2049 > 175.10.0.62.646972160: reply ERR 1448 09:51:05.216579 IP 175.10.0.95.2049 > 175.10.0.62.16784333: reply ERR 1448 09:51:05.216586 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53996197 win 24571 <nop,nop,timestamp 1083651695 57298293> 09:51:05.216701 IP 175.10.0.95.2049 > 175.10.0.62.1678844834: reply ERR 1448 09:51:05.216824 IP 175.10.0.95.2049 > 175.10.0.62.716701787: reply ERR 1448 09:51:05.216831 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 53999093 win 24571 <nop,nop,timestamp 1083651695 57298293> 09:51:05.216947 IP 175.10.0.95.2049 > 175.10.0.62.1331588799: reply ERR 1448 09:51:05.217070 IP 175.10.0.95.2049 > 175.10.0.62.2164270270: reply ERR 1448 09:51:05.217077 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54001989 win 24571 <nop,nop,timestamp 1083651695 57298293> 09:51:05.217193 IP 175.10.0.95.2049 > 175.10.0.62.7307520: reply ERR 1448 09:51:05.217315 IP 175.10.0.95.2049 > 175.10.0.62.201331456: reply ERR 1448 09:51:05.217323 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54004885 win 24571 <nop,nop,timestamp 1083651695 57298293> 09:51:05.217438 IP 175.10.0.95.2049 > 175.10.0.62.891126785: reply ERR 1448 09:51:05.217563 IP 175.10.0.95.2049 > 175.10.0.62.655421: reply ERR 1448 09:51:05.217570 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54007781 win 24571 <nop,nop,timestamp 1083651696 57298293> 09:51:05.217685 IP 175.10.0.95.2049 > 175.10.0.62.1115904: reply ERR 1448 09:51:05.217808 IP 175.10.0.95.2049 > 175.10.0.62.1810654585: reply ERR 1448 09:51:05.217815 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54010677 win 24571 <nop,nop,timestamp 1083651696 57298294> 09:51:05.217935 IP 175.10.0.95.2049 > 175.10.0.62.3817660743: reply ERR 1448 09:51:05.218058 IP 175.10.0.95.2049 > 175.10.0.62.141783552: reply ERR 1448 09:51:05.218064 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54013573 win 24571 <nop,nop,timestamp 1083651696 57298294> 09:51:05.218182 IP 175.10.0.95.2049 > 175.10.0.62.101093888: reply ERR 1448 09:51:05.218304 IP 175.10.0.95.2049 > 175.10.0.62.11927648: reply ERR 1448 09:51:05.218315 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54016469 win 24571 <nop,nop,timestamp 1083651696 57298294> 09:51:05.218426 IP 175.10.0.95.2049 > 175.10.0.62.348889607: reply ERR 1448 09:51:05.218554 IP 175.10.0.95.2049 > 175.10.0.62.101450317: reply ERR 1448 09:51:05.218561 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54019365 win 24571 <nop,nop,timestamp 1083651697 57298294> 09:51:05.218677 IP 175.10.0.95.2049 > 175.10.0.62.2810254248: reply ERR 1448 09:51:05.218800 IP 175.10.0.95.2049 > 175.10.0.62.841483264: reply ERR 1448 09:51:05.218807 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54022261 win 24571 <nop,nop,timestamp 1083651697 57298295> 09:51:05.218923 IP 175.10.0.95.2049 > 175.10.0.62.43745795: reply ERR 1448 09:51:05.219046 IP 175.10.0.95.2049 > 175.10.0.62.1930551515: reply ERR 1448 09:51:05.219054 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54025157 win 24571 <nop,nop,timestamp 1083651697 57298295> 09:51:05.219168 IP 175.10.0.95.2049 > 175.10.0.62.617053963: reply ERR 1448 09:51:05.219292 IP 175.10.0.95.2049 > 175.10.0.62.159516990: reply ERR 1448 09:51:05.219299 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54028053 win 24571 <nop,nop,timestamp 1083651697 57298295> 09:51:05.219414 IP 175.10.0.95.2049 > 175.10.0.62.2390542669: reply ERR 1448 09:51:05.219539 IP 175.10.0.95.2049 > 175.10.0.62.70784577: reply ERR 1448 09:51:05.219546 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54030949 win 24571 <nop,nop,timestamp 1083651698 57298295> 09:51:05.219661 IP 175.10.0.95.2049 > 175.10.0.62.2399374851: reply ERR 1448 09:51:05.219791 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54033845 win 24571 <nop,nop,timestamp 1083651698 57298296> 09:51:05.220037 IP 175.10.0.62.1019 > 175.10.0.95.2049: . ack 54036741 win 24571 <nop,nop,timestamp 1083651698 57298296> 09:51:05.471134 IP 175.10.0.95.2049 > 175.10.0.62.1019: . ack 500424 win 2895 <nop,nop,timestamp 57298695 1083651909> 19667 packets captured 60835 packets received by filter 41168 packets dropped by kernel [root@thing2 ~]#
There's something ugly going on with the network. Take a look at this experiment. I have a bunch of unused local disk space on my thing2 host and I wondered what would happen if I tried copying that large VM disk image from that RHEV Export NFS server to a local disk by hand? So I tried it - see below and then some comments after that. ********************************* [root@thing2 4c352001-ca40-418b-90be-4019f0f6a9e8]# [root@thing2 4c352001-ca40-418b-90be-4019f0f6a9e8]# pwd /rhev/data-center/mnt/175.10.0.95:_home_RHEVexports/f76c1443-10d8-49a0-a140-2c0a2085acbe/images/4c352001-ca40-418b-90be-4019f0f6a9e8 [root@thing2 4c352001-ca40-418b-90be-4019f0f6a9e8]# ls -al total 78125020 drwxr-xr-x 2 vdsm kvm 4096 Dec 6 04:32 . drwxrwx--- 3 vdsm kvm 4096 Dec 6 04:34 .. -rw-r--r-- 1 vdsm kvm 80000000000 Dec 6 04:34 e8d37427-3bbc-46bd-80c5-c69e7c577c27 -rw-r--r-- 1 vdsm kvm 331 Dec 6 04:32 e8d37427-3bbc-46bd-80c5-c69e7c577c27.meta [root@thing2 4c352001-ca40-418b-90be-4019f0f6a9e8]# ls /data/greg [root@thing2 4c352001-ca40-418b-90be-4019f0f6a9e8]# cp e8d37427-3bbc-46bd-80c5-c69e7c577c27 /data/greg *********************************** And my window I pasted in above doing the copying is hung. Looking in other windows, it copied about 204 MB to my local /data/greg directory before hanging. Ctrl/C doesn't work, it's just hung. After a couple minutes, it finally responded to my CTRL/C. From another window, trying: brctl show also hangs. After several minutes, it comes back to a shell after CTRL/C, but never shows me any brctl output. Go figure. Been a long night. I'm going to grab some sleep. - Greg
There's something ugly going on with the network. . . .
And to close the loop on this issue for anyone else trying P2V migrations. Turns out, there were two independent problems that interacted to make a big mess. Problem #1 - I promise never again to set up a 100mb NFS server with gb NFS clients. Apparently, a fast NFS client quickly swamps the slow NFS server with read requests when trying to bulk-copy anything from that NFS server. Curiously, a fast NFS client can write to a slow NFS server just fine. It's **reading** from a slow NFS server that messes it up. Go figure. Problem #2 - See Red Hat Bugzilla number 754331. When my first RHEV host NFS client became hopelessly lost, I rebooted it and it refused to connect with the world after it came back up. After spending 4 days of my life and countless hours of Red Hat support, it turned out that my RHEV host had an extra bogus bridge device named breth0 and that bogus bridge device messed up its communication with the world. After I fixed the bridge issue and we put a gb NIC in the NFS server, I was able to successfully - and quickly - import our P2V'd test VM into our RHEV environment. - Greg
I may be a little closer to the problem. I added a profile named Stylmark-rhev-sparse and used it on my P2V attempts that blew up today. I had been using the profile named Stylmark-rhev-preallocated. I just started up another one with the new 0.8.5.1 CD and P2V version with the Stylmark-rhev-preallocated profile and now it seems to be copying data. Profile definitions below, hopefully not garbled. [root@Fedora16-64P2V log]# cd /etc [root@Fedora16-64P2V etc]# diff virt-v2v.conf virt-v2v.conf-original 38,57d37 < <profile name="Stylmark-rhev-sparse"> < <method>rhev</method> < <storage format='qcow2' allocation='sparse'> < 175.10.0.95:/home/RHEVexports < </storage> < <network type="default"> < <network type="network" name="rhevm"/> < </network> < </profile> < < <profile name="Stylmark-rhev-preallocated"> < <method>rhev</method> < <storage format='raw' allocation='preallocated'> < 175.10.0.95:/home/RHEVexports < </storage> < <network type="default"> < <network type="network" name="rhevm"/> < </network> < </profile> < [root@Fedora16-64P2V etc]# - Greg
Here's some more ugly behavior. The P2V we started several hours ago with 0.8.5.1 looks like it's hung. It copied most of the source server and then apparently decided to go on strike. The diagnostic log file is empty. I'm going to paste in what I see from both the migration server and the Fedora 16 server hosting the NFS service. First the migration server. Notice the ownership on the directory in the Export domain. The migration server shows it as nobody.nobody, while the NFS server shows it as 36.36. What's up with that? And then look at the permissions - that explains the Permission Denied errors. And then because the ownership from the NFS export is somehow messed up, the NFS client / migration server seizes up and here we sit. But how/why would the ownership become messed up most of the way through it? [root@Fedora16-64P2V log]# df -h Filesystem Size Used Avail Use% Mounted on rootfs 47G 4.4G 41G 10% / devtmpfs 2.0G 0 2.0G 0% /dev tmpfs 2.0G 72K 2.0G 1% /dev/shm tmpfs 2.0G 48M 1.9G 3% /run /dev/mapper/vg_fedora1664p2v-lv_root 47G 4.4G 41G 10% / tmpfs 2.0G 48M 1.9G 3% /run tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup tmpfs 2.0G 0 2.0G 0% /media /dev/vda2 485M 81M 379M 18% /boot 175.10.0.95:/home/RHEVexports 212G 66G 136G 33% /tmp/aR_fSTb7EL [root@Fedora16-64P2V log]# cd /tmp/aR_fSTb7EL [root@Fedora16-64P2V aR_fSTb7EL]# ls f76c1443-10d8-49a0-a140-2c0a2085acbe [root@Fedora16-64P2V aR_fSTb7EL]# cd f76c1443-10d8-49a0-a140-2c0a2085acbe -bash: cd: f76c1443-10d8-49a0-a140-2c0a2085acbe: Permission denied [root@Fedora16-64P2V aR_fSTb7EL]# ls -al total 12 drwxr-xr-x. 3 nobody nobody 4096 Dec 6 04:25 . drwxrwxrwt. 16 root root 4096 Dec 12 15:42 .. drwxrwx---. 6 nobody nobody 4096 Dec 12 15:40 f76c1443-10d8-49a0-a140-2c0a2085acbe [root@Fedora16-64P2V aR_fSTb7EL]# ls -al -R .: total 12 drwxr-xr-x. 3 nobody nobody 4096 Dec 6 04:25 . drwxrwxrwt. 16 root root 4096 Dec 12 15:42 .. drwxrwx---. 6 nobody nobody 4096 Dec 12 15:40 f76c1443-10d8-49a0-a140-2c0a2085acbe ls: cannot open directory ./f76c1443-10d8-49a0-a140-2c0a2085acbe: Permission denied [root@Fedora16-64P2V aR_fSTb7EL]# [root@Fedora16-64P2V aR_fSTb7EL]# [root@Fedora16-64P2V aR_fSTb7EL]# ps ax | grep p2v 4893 ? Ss 6:14 /usr/bin/perl /usr/bin/virt-p2v-server 5206 pts/0 S+ 0:00 grep --color=auto p2v [root@Fedora16-64P2V aR_fSTb7EL]# And now the NFS server hosting the Export domain: [root@IT ~]# [root@IT ~]# more /etc/exports /home/RHEVexports *(rw) /home/testnfs *(rw) [root@IT ~]# cd /home/RHEVexports [root@IT RHEVexports]# ls -al -R .: total 12 drwxr-xr-x. 3 36 36 4096 Dec 6 04:25 . drwxr-xr-x. 7 root root 4096 Dec 7 22:19 .. drwxrwx---. 6 36 36 4096 Dec 12 15:40 f76c1443-10d8-49a0-a140-2c0a2085acbe ./f76c1443-10d8-49a0-a140-2c0a2085acbe: total 24 drwxrwx---. 6 36 36 4096 Dec 12 15:40 . drwxr-xr-x. 3 36 36 4096 Dec 6 04:25 .. drwxrwx---. 2 36 36 4096 Dec 6 11:02 dom_md drwxrwx---. 2 36 36 4096 Dec 9 16:49 images drwxr-x---. 4 36 36 4096 Dec 5 20:30 master drwx------. 3 36 36 4096 Dec 12 15:40 v2v.3R3Eo5HD ./f76c1443-10d8-49a0-a140-2c0a2085acbe/dom_md: total 16 drwxrwx---. 2 36 36 4096 Dec 6 11:02 . drwxrwx---. 6 36 36 4096 Dec 12 15:40 .. -rw-rw----. 1 36 36 8388608 Dec 5 20:27 ids -rw-rw----. 1 36 36 16777216 Dec 5 20:27 inbox -rw-rw----. 1 36 36 2147483648 Dec 6 11:02 leases -rw-rw----. 1 36 36 393 Dec 6 11:02 metadata -rw-rw----. 1 36 36 16777216 Dec 5 20:27 outbox ./f76c1443-10d8-49a0-a140-2c0a2085acbe/images: total 8 drwxrwx---. 2 36 36 4096 Dec 9 16:49 . drwxrwx---. 6 36 36 4096 Dec 12 15:40 .. ./f76c1443-10d8-49a0-a140-2c0a2085acbe/master: total 16 drwxr-x---. 4 36 36 4096 Dec 5 20:30 . drwxrwx---. 6 36 36 4096 Dec 12 15:40 .. drwxrwx---. 2 36 36 4096 Dec 5 20:30 tasks drwxrwx---. 2 36 36 4096 Dec 9 16:49 vms ./f76c1443-10d8-49a0-a140-2c0a2085acbe/master/tasks: total 8 drwxrwx---. 2 36 36 4096 Dec 5 20:30 . drwxr-x---. 4 36 36 4096 Dec 5 20:30 .. ./f76c1443-10d8-49a0-a140-2c0a2085acbe/master/vms: total 8 drwxrwx---. 2 36 36 4096 Dec 9 16:49 . drwxr-x---. 4 36 36 4096 Dec 5 20:30 .. ./f76c1443-10d8-49a0-a140-2c0a2085acbe/v2v.3R3Eo5HD: total 12 drwx------. 3 36 36 4096 Dec 12 15:40 . drwxrwx---. 6 36 36 4096 Dec 12 15:40 .. drwxr-xr-x. 2 36 36 4096 Dec 12 15:40 ac8f73a6-6515-4797-91a4-fd2fc4bc4f56 ./f76c1443-10d8-49a0-a140-2c0a2085acbe/v2v.3R3Eo5HD/ac8f73a6-6515-4797-91a4-fd2fc4bc4f56: total 67043348 drwxr-xr-x. 2 36 36 4096 Dec 12 15:40 . drwx------. 3 36 36 4096 Dec 12 15:40 .. -rw-r--r--. 1 36 36 80000000000 Dec 12 17:19 7dccc45e-a86e-4056-b567-db36786cc03e [root@IT RHEVexports]# I wonder if that P2V would pick back up again if I liberalized the permissions by hand? - Greg
participants (2)
-
Greg Scott -
Richard W.M. Jones