Re: [Libguestfs] P2Vs seem to require a very robust Ethernet
I did one with ntbackup and restore a few months ago and ran into problems with the virtual cpu not matching the virtual bios. That one was in a libvirt environment and that VM will retire soon. But with the current project, these VMs will be around a while. Sent from my Droid Charge on Verizon 4GLTE Matthew Booth <mbooth@redhat.com> wrote: On 12/02/2011 03:12 AM, Greg Scott wrote:
Hey Matt -
With a new release of virt-v2v a while away, what if we were to try a P2V by hand? I'm thinking of maybe provisioning a Windows RHEV VM, doing an ntbackup of the source physical machine and doing a bare metal restore on the VM. I've done it that way before with mixed results.
We have that hack I put together to load the viostor block driver stuff on the source machine so the target VM will boot. Are there any HAL kinds of things or other hardware/firmware issues we need to also be thinking about?
Windows mostly fixes itself. If it boots and you don't see any missing drivers in Device Manager, you're probably good to go. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490
Our track record for P2Vs isn't very good so far; we've had one run to completion in about 8 or 9 tries so far. We've tried source systems with different versions of Windows, all but one failed. The one success was a test Windows XP machine with one hard drive. We boot a Windows system from the CD, start up a P2V, it runs for an apparently random length of time, then the P2V percentage just stops somewhere. The other day was 13 percent. Today it's 8 percent. Eventually, virt-p2v-server on the conversion server also stops and I see a message in the log file that says, " virt-v2v: Error receiving data:". Is there anything we can look at from an alternate console on the source server booted from that CD? Thanks - Greg
After 5 or 6 hours, we aborted today's attempt, disconnected the 2nd HDD (sdb) from the original host, and started up a new one. Since all we have to go on are guesses - there was a bug with the Windows virtio network driver where the driver would go offline during bulk copies. I bring this up because our P2V conversion server is a Fedora VM, probably using the Linux virtio driver. Was there a similar bug with the Linux VirtIO driver? One possible sequence of events: - Data flies fast and furious in and out of the conversion server VM to the NFS server. - The conversion server VM NIC goes offline. - The source host eventually hangs or dies. - virt-p2v-server eventually gives up and writes a cryptic log message. Another possible sequence of events: - The source server starts copying. - The source server runs into a problem and either hangs or dies. - Virt-p2v-server on the conversion server eventually times out waiting for data. What do you think about trying a physical conversion server, possibly using itself as the NFS Server and RHEV Export domain? - Greg -----Original Message----- From: libguestfs-bounces@redhat.com [mailto:libguestfs-bounces@redhat.com] On Behalf Of Greg Scott Sent: Friday, December 02, 2011 3:18 PM To: Matthew Booth; Richard W.M. Jones; libguestfs@redhat.com Cc: Fredy Hernandez; Joey Bertalan Subject: Re: [Libguestfs] P2Vs seem to require a very robust Ethernet Our track record for P2Vs isn't very good so far; we've had one run to completion in about 8 or 9 tries so far. We've tried source systems with different versions of Windows, all but one failed. The one success was a test Windows XP machine with one hard drive. We boot a Windows system from the CD, start up a P2V, it runs for an apparently random length of time, then the P2V percentage just stops somewhere. The other day was 13 percent. Today it's 8 percent. Eventually, virt-p2v-server on the conversion server also stops and I see a message in the log file that says, " virt-v2v: Error receiving data:". Is there anything we can look at from an alternate console on the source server booted from that CD? Thanks - Greg _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
I like it, we have the space. alternatively i build a physical fedora 16 box with 500 gb of space... I can get on it tomorrow night -----Original Message----- From: Greg Scott [mailto:GregScott@Infrasupport.com] Sent: Fri 12/2/2011 5:57 PM To: Greg Scott; Matthew Booth; Richard W.M. Jones; libguestfs@redhat.com Cc: Fredy Hernández; Joey Bertalan Subject: RE: [Libguestfs] P2Vs seem to require a very robust Ethernet After 5 or 6 hours, we aborted today's attempt, disconnected the 2nd HDD (sdb) from the original host, and started up a new one. Since all we have to go on are guesses - there was a bug with the Windows virtio network driver where the driver would go offline during bulk copies. I bring this up because our P2V conversion server is a Fedora VM, probably using the Linux virtio driver. Was there a similar bug with the Linux VirtIO driver? One possible sequence of events: - Data flies fast and furious in and out of the conversion server VM to the NFS server. - The conversion server VM NIC goes offline. - The source host eventually hangs or dies. - virt-p2v-server eventually gives up and writes a cryptic log message. Another possible sequence of events: - The source server starts copying. - The source server runs into a problem and either hangs or dies. - Virt-p2v-server on the conversion server eventually times out waiting for data. What do you think about trying a physical conversion server, possibly using itself as the NFS Server and RHEV Export domain? - Greg -----Original Message----- From: libguestfs-bounces@redhat.com [mailto:libguestfs-bounces@redhat.com] On Behalf Of Greg Scott Sent: Friday, December 02, 2011 3:18 PM To: Matthew Booth; Richard W.M. Jones; libguestfs@redhat.com Cc: Fredy Hernandez; Joey Bertalan Subject: Re: [Libguestfs] P2Vs seem to require a very robust Ethernet Our track record for P2Vs isn't very good so far; we've had one run to completion in about 8 or 9 tries so far. We've tried source systems with different versions of Windows, all but one failed. The one success was a test Windows XP machine with one hard drive. We boot a Windows system from the CD, start up a P2V, it runs for an apparently random length of time, then the P2V percentage just stops somewhere. The other day was 13 percent. Today it's 8 percent. Eventually, virt-p2v-server on the conversion server also stops and I see a message in the log file that says, " virt-v2v: Error receiving data:". Is there anything we can look at from an alternate console on the source server booted from that CD? Thanks - Greg _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs Fredy Hernandez Systems Administrator 763-574-8734 Office hernandez@stylmark.com
Here's another nugget of info. Watching top on my conversion server, I consistently see roughly 50% id and 50% wa. It varies second by second by a percent or two but it's generally around 50-50. The "wa" state means waiting for an I/O to complete so it seems like something is taking a **very** long time to complete IOs. My guess is, something is stalling virt-p2v-server. I wonder if there's a way to drill down into an individual process the way we used to do it with OpenVMS? Every once-in-a-while, virt-p2v-server will jump up to the top of the top list. The current P2V has been running for about 6 hours so far and I don't know what the state looks like on the source server console. The logfile on the conversion server is still blank. Watching the Export domain with my RHEV Manager, I see this P2V has used about 28 GB of space on the NFS server so far. On the source server - how do we know the NIC speed and duplex? I guess we can look at the switch port from the switch's point of view, but how would we look at it and/or set it from the host point of view? - Greg
14 hours after starting it up - about 41 GB done. The top utility still shows about 50-50 between id and wa. Assuming virt-p2v-server is stalling, I wonder how to find if it's stalling waiting on network I/O from the source server or waiting on network I/O to the NFS server? - Greg
And some more clues. Apparently, our virt-p2v-server is dropping lots of received packets. I wonder why? [root@Fedora16-64P2V log]# netstat -i Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 347387803 0 29625 0 134464820 0 0 0 BMRU lo 16436 0 1088 0 0 0 1088 0 0 0 LRU virbr0 1500 0 0 0 0 0 0 0 0 0 BMU [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# netstat -i Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 347393913 0 29630 0 134465886 0 0 0 BMRU lo 16436 0 1088 0 0 0 1088 0 0 0 LRU virbr0 1500 0 0 0 0 0 0 0 0 0 BMU [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# netstat -i Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 347394009 0 29635 0 134465895 0 0 0 BMRU lo 16436 0 1088 0 0 0 1088 0 0 0 LRU virbr0 1500 0 0 0 0 0 0 0 0 0 BMU [root@Fedora16-64P2V log]#
I found a presentation from IBM that shows how to do a bunch of drilldown stuff: http://linuxvm.org/present/SHARE103/S9303ob.pdf It mentions a utility named strace that shows system calls. I ran one for a couple seconds against the running virt-p2v-server - here are the results: [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# ps ax | grep p2v 23339 ? Ds 2:44 /usr/bin/perl /usr/bin/virt-p2v-server 24048 pts/0 S+ 0:00 grep --color=auto p2v [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# strace -p 23339 -tt -T -o greg.txt Process 23339 attached - interrupt to quit ^CProcess 23339 detached [root@Fedora16-64P2V log]# more greg.txt 06:53:04.258833 write(5, "R\0e\0q\0u\0e\0s\0t\0:\0 \0N\0u\0l\0l\0;\0 \0 \0"..., 8192) = 8 192 <0.000107> 06:53:04.259246 write(5, "I\0n\0s\0t\0a\0l\0l\0e\0d\0:\0 \0N\0u\0l\0l\0;\0"..., 8192) = 8 192 <0.000100> 06:53:04.259503 write(5, "e\0.\0x\0m\0l\0_\0004\0005\0003\0009\0004\0_\0e\0n\0u\0_\0"..., 8192) = 8192 <0.000126> 06:53:04.259821 write(5, "m\0p\0o\0n\0e\0n\0t\0:\0 \0_\0_\0V\0W\0D\0_\0W\0"..., 8192) = 8 192 <0.227877> 06:53:04.487853 write(5, "n\0t\0:\0 \0_\0_\0_\0S\0Q\0L\0D\0B\0G\0_\0004\0008\0"..., 8192) = 8192 <0.000105> 06:53:04.488107 write(5, "P\0L\0A\0T\0E\0S\0_\0x\0008\0006\0_\0e\0n\0u\0.\0003\0"..., 819 2) = 8192 <0.000101> 06:53:04.488357 write(5, "_\0A\0005\0003\0006\0_\0000\0000\0009\0000\0002\0007\0008\0A\00 01\0B\0"..., 8192) = 8192 <0.000101> 06:53:04.488664 write(5, "3\0002\0_\0_\0_\0_\0_\0x\0008\0006\0.\0003\0006\0004\0003\0002\ 0"..., 8192) = 8192 <0.228029> 06:53:04.716874 write(5, "A\0005\0003\0006\0_\0000\0000\0009\0000\0002\0007\0008\0A\0001\ 0B\0B\0"..., 8192) = 8192 <0.000112> 06:53:04.717150 write(5, " \0 \0 \0A\0c\0t\0i\0o\0n\0:\0 \0N\0u\0l\0l\0\r\0"..., 8192) = 8192 <0.000108> 06:53:04.717409 write(5, "\r\0\n\0M\0S\0I\0 \0(\0s\0)\0 \0(\0004\0004\0:\0F\0000\0"..., 8 192) = 8192 <0.000112> 06:53:04.717683 write(5, " \0(\0s\0)\0 \0(\0004\0004\0:\0F\0000\0)\0 \0[\0002\0002\0"..., 8192) = 8192 <0.227951> 06:53:04.945788 write(5, "S\0I\0 \0(\0s\0)\0 \0(\0004\0004\0:\0F\0000\0)\0 \0[\0"..., 819 2) = 8192 <0.000105> 06:53:04.946041 write(5, ")\0 \0(\0004\0004\0:\0F\0000\0)\0 \0[\0002\0002\0:\0003\0000\0" ..., 8192) = 8192 <0.000100> 06:53:04.946288 write(5, ".\0003\0006\0004\0003\0002\0003\0006\0F\0_\0F\0C\0007\0000\0_\0 001\0"..., 8192) = 8192 <0.000104> 06:53:04.946540 write(5, "n\0g\0_\0_\0_\0V\0S\0D\0e\0b\0u\0g\0_\0R\0e\0g\0"..., 8192) = 8 192 <0.229116> 06:53:05.175810 write(5, "I\0n\0s\0t\0a\0l\0l\0e\0d\0:\0 \0N\0u\0l\0l\0;\0"..., 8192) = 8 192 <0.000105> 06:53:05.176062 write(5, "_\0M\0i\0c\0r\0o\0s\0o\0f\0t\0_\0V\0C\0009\0000\0_\0"..., 8192) = 8192 <0.000109> 06:53:05.176319 write(5, "s\0_\0R\0e\0s\0_\0r\0u\0s\0_\0004\0002\0005\0003\0.\0003\0"..., 8192) = 8192 <0.000120> 06:53:05.176593 write(5, "l\0e\0d\0:\0 \0N\0u\0l\0l\0;\0 \0 \0 \0R\0e\0q\0"..., 8192) = 8 192 <0.228033> 06:53:05.404780 write(5, "u\0l\0l\0\r\0\n\0M\0S\0I\0 \0(\0s\0)\0 \0(\0004\0004\0"..., 819 2) = 8192 <0.000106> 06:53:05.405033 write(5, "l\0;\0 \0 \0 \0R\0e\0q\0u\0e\0s\0t\0:\0 \0N\0u\0"..., 8192) = 8 192 <0.000101> 06:53:05.405280 write(5, "5\0009\0000\0000\0_\0e\0n\0u\0_\0x\0008\0006\0.\0003\0006\0004\ 0"..., 8192) = 8192 <0.000104> 06:53:05.405641 write(5, "b\0_\0002\0003\0002\0003\0000\0.\0003\0006\0004\0003\0002\0003\ 0006\0F\0"..., 8192) = 8192 <0.230937> 06:53:05.636740 write(5, "e\0s\0t\0:\0 \0L\0o\0c\0a\0l\0;\0 \0 \0 \0A\0c\0"..., 8192) = 8 192 <0.000106> 06:53:05.636996 write(5, "o\0n\0:\0 \0N\0u\0l\0l\0\r\0\n\0M\0S\0I\0 \0(\0s\0"..., 8192) = 8192 <0.000106> 06:53:05.637250 write(5, "t\0:\0 \0_\0_\0K\0B\0009\0005\0004\0003\0004\0005\0.\0I\0n\0".. ., 8192) = 8192 <0.000104> 06:53:05.637512 write(5, "A\0005\0003\0006\0_\0000\0000\0009\0000\0002\0007\0008\0A\0001\ 0B\0B\0"..., 8192) = 8192 <0.233094> 06:53:05.870776 write(5, "R\0i\0b\0b\0o\0n\0_\0C\0a\0t\0e\0g\0o\0r\0y\0C\0"..., 8192) = 8 192 <0.000108> 06:53:05.871042 write(5, "S\0I\0 \0(\0s\0)\0 \0(\0004\0004\0:\0F\0000\0)\0 \0[\0"..., 819 2) = 8192 <0.000101> 06:53:05.871296 write(5, "_\0I\0n\0c\0l\0u\0d\0e\0_\0B\0l\0a\0c\0k\0_\0M\0"..., 8192) = 8 192 <0.000106> 06:53:05.871609 write(5, "(\0s\0)\0 \0(\0004\0004\0:\0F\0000\0)\0 \0[\0002\0002\0:\0"..., 8192) = 8192 <0.228984> 06:53:06.100767 write(5, "e\0d\0:\0 \0A\0b\0s\0e\0n\0t\0;\0 \0 \0 \0R\0e\0"..., 8192) = 8 192 <0.000109> 06:53:06.101038 write(5, "l\0e\0d\0:\0 \0L\0o\0c\0a\0l\0;\0 \0 \0 \0R\0e\0"..., 8192) = 8 192 <0.000105> 06:53:06.101293 write(5, "a\0l\0;\0 \0 \0 \0A\0c\0t\0i\0o\0n\0:\0 \0N\0u\0"..., 8192) = 8 192 <0.000103> 06:53:06.101553 write(5, "n\0t\0;\0 \0 \0 \0R\0e\0q\0u\0e\0s\0t\0:\0 \0L\0"..., 8192 <unf inished ...> [root@Fedora16-64P2V log]#
Trying strace again didn't work very well - I wonder if it was stuck waiting for a network I/O to finish? [root@Fedora16-64P2V log]# ps ax | grep p2v 23339 ? Ds 2:44 /usr/bin/perl /usr/bin/virt-p2v-server 24048 pts/0 S+ 0:00 grep --color=auto p2v [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# strace -p 23339 -tt -T Process 23339 attached - interrupt to quit ^C^C^C^C^C^C^C^C^C^C^C^C ^Z [1]+ Stopped strace -p 23339 -tt -T [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# ps ax | grep p2v 23339 ? Ds 2:45 /usr/bin/perl /usr/bin/virt-p2v-server 24054 pts/0 R+ 0:00 grep --color=auto p2v [root@Fedora16-64P2V log]# ps ax | grep strace 24052 pts/0 T 0:00 strace -p 23339 -tt -T 24056 pts/0 S+ 0:00 grep --color=auto strace [root@Fedora16-64P2V log]# attach -h bash: attach: command not found... [root@Fedora16-64P2V log]# kill -9 24052 [1]+ Killed strace -p 23339 -tt -T [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# ps ax | grep p2v 23339 ? Ds 2:45 /usr/bin/perl /usr/bin/virt-p2v-server 24065 pts/0 S+ 0:00 grep --color=auto p2v [root@Fedora16-64P2V log]#
Trying strace against virt-p2v-server again, this time to stdout - A bunch of writes, then another hang. I annotated this one a little bit. That hang went for more than 3 **minutes** from one write to the next. Combined with a top display showing the overall system is spending about 1/2 its time waiting for I/Os to complete, does this tell us anything? . . . 07:04:10.643462 write(5, "\0\0\0\0\20\220\362\10\0\0\0\377\376\377R\\\0\\\0P\0e\0t\0a\0g\0o\0n"..., 8192) = 8192 <0.227277> 07:04:10.870894 write(5, "\1\0\0\0\0\0Iw\330@\377\376\377\10M\0o\0d\0i\0f\0i\0e\0d\0\5\0"..., 8192) = 8192 <0.000032> 07:04:10.871052 write(5, "\0\7\201\377\376\377\0062\0001\0001\0001\0002\0003\0\2\0\0\210\360\2617\0G\4\4K\6"..., 8192) = 8192 <0.000035> 07:04:10.871211 write(5, "-\vd\216^\7\10\266\274\330\214}N\2\2xN\263i,\271\5\242\242w\374v\357\315\233O\34"..., 8192) = 8192 <0.000033> 07:04:10.871423 write(5, "\377\377\377\377\377\377\377\377L\377\0\0\377\377\377\377\377\377\377\377\377\377L\377\0\0\377\377\377\377\377\377"..., 8192) = 8192 <0.227303> 07:04:11.098886 write(5, "\2\0\0\5\7\t\366\377\366\0\2\0\4\t\2\0\377\377\377\377\377\377\24\377\0\0\377\377\355\377\2\0"..., 8192) = 8192 <0.000034> 07:04:11.099092 write(5, "\1\0\377\377\345\377\0\0\377\377\377\377\377\377\34\377\2\0\31\7\3\0\26\7\10\0\2\366\0\4\0\7"..., 8192) = 8192 <0.000031> 07:04:11.099294 write(5, "\377\377L\377\0\0\377\377\377\377\377\377\377\377\377\377L\377\0\0\377\377\377\377\377\377\377\377\377\377L\377"..., 8192) = 8192 <0.000036> 07:04:11.099503 write(5, "N\0D\0_\0P\0I\0N\0S\0\20\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0"..., 8192 (I inserted blank space here - note the time indexes - more than 3 minutes when nothing happened.) ) = 8192 <195.972190> 07:07:27.071878 write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192 <0.000100> 07:07:27.072111 write(5, "\35\202\0\0\364\201\305\2\247\1\0\0\240\336\2469\362\201\0\0\364\201\305\2\307\1\0\0\365R\2659"..., 8192) = 8192 <0.000046> 07:07:27.072269 write(5, "\5\0\0\0\35\202\0\0\364\201\366\1\247\1\0\0\240\336\2469\362\201\0\0\364\201\366\1\307\1\0\0"..., 8192) = 8192 <0.000049> 07:07:27.072420 write(5, "\364\201\366\1\247\1\0\0\240\336\2469\362\201\0\0\364\201\366\1\307\1\0\0\365R\2659\2\0\0\0"..., 8192) = 8192 <0.226265> 07:07:27.298847 write(5, "?\32p\303u\371\234\355?\2\352%S\253\374\354?\314\314\235\373&#\257\2770u.\301L\21\202"..., 8192) = 8192 <0.000106> 07:07:27.299069 write(5, "\357\345\313\303\260\277\177\23\256\256B\370\376]\235_3\263\312U\t4' \0006g\353\312\4\10$"..., 8192) = 8192 <0.000100> 07:07:27.299269 write(5, "@ 1\1\0010\261\1\327]\2\4\10\20 @\200\200\0h\16\20 @\200\0\1\2\4\22\23\20"..., 8192) = 8192 <0.000100> 07:07:27.299470 write(5, "F%u\2\212?\1\0\0\0\0\0\0\0\377\376\377\0\377\376\377\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192 <0.227184> 07:07:27.526791 write(5, "\303&\234\334\314\345?\0\0\0\0\0\0\0\0\347\260x\307\207\324\356?\3407\200et%\321?\330"..., 8192) = 8192 <0.000051> 07:07:27.526952 write(5, "\6\2K\0\0\0x\235pI\5\0\0\0\377\377\377\377\0\0\0\0\3\202\0\0\364\201\6\2\366\0"..., 8192) = 8192 <0.000052> 07:07:27.527156 write(5, "\300\300\240\0\360\373\377\0\244\240\240\0\200\200\200\0\0\0\377\0\0\377\0\0\0\377\377\0\377\0\0\0"..., 8192) = 8192 <0.000057> 07:07:27.527361 write(5, "}\377\2\0T\7\2\0\1\7\1\0\2\7\5\0\4\366\2\0\2\7\1\10\2\366\3\377\1\t\2\0"..., 8192^C <unfinished ...> Process 23339 detached [root@Fedora16-64P2V log]#
Watching strace -p 23339 -tt -T in another window, where 23339 is the PID of our virt-v2v-server process - reads just machine gun past. But now it's doing a bunch of writes and they blip past kind of like individual shotgun blasts. OK, lousy metaphor but you get the idea. And some of the writes hang for a while, like the one I caught earlier. In fact, I just grabbed another couple of hanging writes - see below. I inserted a bunch of blank space at each hang. And now that we have some footprints, see an action plan below that. . . . 10:58:04.647333 write(5, "*\215N\367\273\377\1\0\0\323\343\v\323\213\312\301\371\10\210\17GN\301\342\10\203\376\v~2\203\356"..., 8192) = 8192 <0.233197> 10:58:04.880702 write(5, "\310\323\343\3\323\213\336\203\300\10\353\10\205\300\17\214\3\2\0\0\212\2174\354\2\0\204\311\2134\275X"..., 8192) = 8192 <0.000043> 10:58:04.880935 write(5, "\306\213\3513\3523\357\301\311\2\3\305\213t$`\213l$h3\365\213\254$\200\0\0\0003\365\213"..., 8192) = 8192 <0.000034> 10:58:04.881103 write(5, "\n\213\302\203\360\377\3\373\213i8\v\3073\303\3\365\5\247#\224\253\3\360\301\306\17\213\303\203\360\377"..., 8192) = 8192 <0.000033> 10:58:04.881269 write(5, "\311\302\34\0\314\314\314\314\314\314\213\377U\213\354\213M\20\213E\fS3\333\210\10@C\205\311\17\216"..., 8192) = 8192 <0.233241> 10:58:05.114669 write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192 <0.000038> 10:58:05.114932 write(5, "WithTag\0N\0ExFreePoolWithTag\0\265\5st"..., 8192) = 8192 <0.000035> 10:58:05.115095 write(5, ".rdata\0\0$\2\0\0\200C\0\0\200\2\0\0\200C\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192 <0.000039> 10:58:05.115306 write(5, "\377\377\351\263\376\377\377j\0j\1\215\206\264\2\0\0P\377\25\10D\1\0\351\254\376\377\377\215\216\234"..., 8192) = 8192 <0.233190> 10:58:05.348659 write(5, "\2\2111^[_]\302(\0\314\314\314\314\314\314\377%\264C\1\0\314\314\314\314\314\314\377%\270C"..., 8192) = 8192 <0.000038> 10:58:05.348884 write(5, "\356\1\0\0\0\25\0\0\0\2\0\0\0\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0@\0\0H"..., 8192) = 8192 <0.000035> 10:58:05.349055 write(5, "P\0\0\0\32\0\0\0\0\5\0\0 \3\0\0\0\17\0\0\1\0\0\0\30\0\0\0<\0\0\0"..., 8192) = 8192 <0.000082> 10:58:05.349311 write(5, "u\30\377u\24\377u\20S\377u\10\350\375\30\1\0;\307\211E$\17\204\204\2\0\0009}\364\17"..., 8192 ) = 8192 <63.979352> 10:59:09.328867 write(5, "\1\0\0\213K\4\213E\350;H\ft\r\213M\10\366A\34\1\17\205L\1\0\0\377u\10P\350"..., 8192) = 8192 <0.000070> 10:59:09.329110 write(5, "WP\350s\331\0\0\211\2050\377\377\377\213M\374\213\2050\377\377\377_^[\350\333\324\0\0\311\302"..., 8192) = 8192 <0.000048> 10:59:09.329268 write(5, "\354\f\213\r\250Q\367\277SV\301\341\5\213\221\300Q\367\277\213\201\310Q\367\277W\213}\10\213w$"..., 8192) = 8192 <0.000050> 10:59:09.329422 write(5, "\17\204\244\0\0\0\213G\4;C\fu\31\377u\20\215E\350P\350\354p\377\377j\0S\350\212\234"..., 8192) = 8192 <0.536185> 10:59:09.865891 write(5, "\300\213E\274\3\310\211M\334\213\n;\371}\2\213\317\3\310\213B\3749E\304\211M\344\213M\304\177"..., 8192) = 8192 <0.000169> 10:59:09.866274 write(5, "\277\211\204\21\354\27\1\0\2418@\367\277\203\274\1\354\27\1\0\0\17\204\362\0\0\0\213O\f#\313"..., 8192) = 8192 <0.000163> 10:59:09.866663 write(5, "\300\212g\367\212G\370f\205\300t\3\200\n\0043\300\212g\371\212G\372f\205\300t\3\200\n\0103"..., 8192) = 8192 <0.000168> 10:59:09.867043 write(5, "\0w\n\306\2\320\212\331B\376\313\353\351\306\2\367Bf\211\nBB\213]\364\210Z\3\301\353\10\210"..., 8192 ) = 8192 <189.274151> 11:02:19.141412 write(5, "\0\4\0\4\2\6\2\6\0\4\0\4\2\6\2\6\1\5\1\5\3\7\3\7\1\5\1\5\3\7\3\7"..., 8192) = 8192 <0.233716> 11:02:19.375339 write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192 <0.000288> 11:02:19.375800 write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0.text\0\0\0 4\t\0\0\20\0\0"..., 8192) = 8192 <0.000269> 11:02:19.376243 write(5, "L\2\0\0\205\300t\23Pj\0\377\25|V\t\20\307\206L\2\0\0\0\0\0\0\213\206P\2\0"..., 8192) = 8192 <0.000279> 11:02:19.376693 write(5, "\0\0\302\10\0j\0S\215L$ Q\213\317\350\234\365\377\377\213\360\205\366|\305\213\317\350\317\345\377"..., 8192) = 8192 <0.234436> 11:02:19.611311 write(5, "U\213\354j\377h\373\332\10\20d\241\0\0\0\0Pd\211%\0\0\0\0\203\354\24S\213]\20V"..., 8192) = 8192 <0.000288> 11:02:19.611772 write(5, "\3\0\0\213\317\306D$\34\4\350i\237\7\0\307\7\24\371\t\20\215\276h\3\0\0\213\317\306D$"..., 8192) = 8192 <0.000295> 11:02:19.612241 write(5, "\374\353\0023\300\213P,\271\1\0\0\0;\321t\r\211H,\213@\10\213\20QP\377R\f_3"..., 8192) = 8192 <0.000265> 11:02:19.612677 write(5, "\213\6W\213\316\307F,\270\2\0\0\377P8\215L$\20Q3\333h\314\263\t\20\211\\$\30\377"..., 8192) = 8192 <0.232419> 11:02:19.845273 write(5, "V\213\361\213Fd\205\300t&\213F`\205\300t\7P\377\25\314V\t\20\213FhP\377\25\330R"..., 8192) = 8192 <0.000290> . . . Based on the evidence we've been able to uncover, it feels to me like we're having a problem writing to the NFS server. Here's why I think so. virt-p2v-server pretty much reads blocks from the source server and writes to the Export NFS share - right? Reads are fast and furious, writes are very slow and sometimes even hang for ***minutes*** at a time. If my hunch is right, then it seems to me we either have an issue with the virtual NIC on our Fedora16-64P2V VM, or the NFS server, or the network in-between. The source server isn't part of the picture because we've tried several source servers and the behavior is the same for every one. We've simplified the network as much as we can. We have a brand new HP Procurve 2910al-24G and all the players are directly connected to that switch. So I think we can eliminate the network since all the transactions travel over the same switch, just to/from different sources and destinations. That leaves either the conversion server or NFS server as culprits. So here's what I'd like to try: Let's eliminate the Storagetek NFS Server first. Let's build up that 500 GB box Fredy mentioned as a physical F16 system and set it up as an NFS server. Change the RHEV Export domain to point to this new NFS server and try some P2Vs using the same Fedora VM as our conversion server. Observe the results with strace and overall. If this doesn't change the symptoms, then let's add the virt-v2v stuff to that 500 GB Fedora system and we'll use it both as a conversion server and NFS server and try again. And finally, to round out the testing, if we still have energy left, we can set up the Storagetek NFS server again as the RHEV export domain and try some more P2Vs with that 500 GB physical Fedora system to the Storagetek. - Greg
40+ hour after starting, that P2V is **still** running and the logfile is 0 length. But curiously, the RHEV Export storage it wants has not increased in about the past 12+ hours. The top display is now showing 99%+ idle. Could virt-p2v-server be stuck in a loop of some kind? - Greg
...And a few seconds after clicking "Send" it finally went up by one more GB. As of right now, we're at 89 total GB used in the RHEV Exports storage. The ENG-BACKUP we did a while ago uses 38 GB, and there's some overhead, so this P2V must be around 45 - 50+ GB used so far. - Greg -----Original Message----- From: libguestfs-bounces@redhat.com [mailto:libguestfs-bounces@redhat.com] On Behalf Of Greg Scott Sent: Sunday, December 04, 2011 6:57 AM To: Matthew Booth; Richard W.M. Jones; libguestfs@redhat.com Cc: Joey Bertalan Subject: Re: [Libguestfs] P2Vs seem to require a very robust Ethernet 40+ hour after starting, that P2V is **still** running and the logfile is 0 length. But curiously, the RHEV Export storage it wants has not increased in about the past 12+ hours. The top display is now showing 99%+ idle. Could virt-p2v-server be stuck in a loop of some kind? - Greg _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
Well now this is very strange. That P2V we started up Friday afternoon crossed into hour 52 a few minutes ago. But in the past 3 1/2 hours, it hasn't used one more byte of disk space in the NFS Export service. But then after firing up an strace just now, it started using space again. What's up with that? [root@Fedora16-64P2V log]# date Sun Dec 4 15:52:22 CST 2011 [root@Fedora16-64P2V log]# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 49058108 3944064 42622040 9% / devtmpfs 2014608 0 2014608 0% /dev tmpfs 2024924 444 2024480 1% /dev/shm tmpfs 2024924 48480 1976444 3% /run /dev/mapper/vg_fedora1664p2v-lv_root 49058108 3944064 42622040 9% / tmpfs 2024924 48480 1976444 3% /run tmpfs 2024924 0 2024924 0% /sys/fs/cgroup tmpfs 2024924 0 2024924 0% /media /dev/vda2 495844 82656 387588 18% /boot 175.10.0.45:/EXPORTS/ 1055113408 93131296 961982112 9% /tmp/UBKmK8QbfM [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# date Sun Dec 4 19:25:52 CST 2011 [root@Fedora16-64P2V log]# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 49058108 3944084 42622020 9% / devtmpfs 2014608 0 2014608 0% /dev tmpfs 2024924 444 2024480 1% /dev/shm tmpfs 2024924 48480 1976444 3% /run /dev/mapper/vg_fedora1664p2v-lv_root 49058108 3944084 42622020 9% / tmpfs 2024924 48480 1976444 3% /run tmpfs 2024924 0 2024924 0% /sys/fs/cgroup tmpfs 2024924 0 2024924 0% /media /dev/vda2 495844 82656 387588 18% /boot 175.10.0.45:/EXPORTS/ 1055113408 93131296 961982112 9% /tmp/UBKmK8QbfM [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# ps ax | grep p2v 23339 ? Ts 3:13 /usr/bin/perl /usr/bin/virt-p2v-server 25668 pts/0 S+ 0:00 grep --color=auto p2v [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# strace -p 23339 -tt -T . . . 19:27:20.968661 write(5, "a\22-\7 \2\1\16\21\211\305\v\7\4\22\202\361\22\202\365\35\16\10\5 \0\22\211\315\6 \1"..., 8192) = 8192 <0.000029> 19:27:20.968921 write(5, "\22-\6 \2\1\35\16\10\r\7\6\22\201e\16\16\35\16\22a\22-\26\7\f\16\22\210 \35\16"..., 8192) = 8192 <0.000027> 19:27:20.969187 write(5, "\16\n\7\4\22\203\261\10\22a\22-#\7\t\22\203\234\22\207\360\10\10\22\207\350\25\22\201\225\1\22"..., 8192) = 8192 <0.000026> 19:27:20.969441 write(5, "\0\0\36\1\0\31MasterPage_MasterPageFile\0"..., 8192) = 8192 <0.000114> 19:27:20.969917 write(5, "\16\16\34\22\224\10\22\204-\22]\22-\22]\35\34\22-\23\7\10\22\203\235\16\22\201i\2\22\204"..., 8192^C <unfinished ...> Process 23339 detached [root@Fedora16-64P2V log]# date Sun Dec 4 19:31:35 CST 2011 [root@Fedora16-64P2V log]# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 49058108 3944084 42622020 9% / devtmpfs 2014608 0 2014608 0% /dev tmpfs 2024924 444 2024480 1% /dev/shm tmpfs 2024924 48480 1976444 3% /run /dev/mapper/vg_fedora1664p2v-lv_root 49058108 3944084 42622020 9% / tmpfs 2024924 48480 1976444 3% /run tmpfs 2024924 0 2024924 0% /sys/fs/cgroup tmpfs 2024924 0 2024924 0% /media /dev/vda2 495844 82656 387588 18% /boot 175.10.0.45:/EXPORTS/ 1055113408 93135072 961978336 9% /tmp/UBKmK8QbfM [root@Fedora16-64P2V log]#
Our P2V conversion server is a RHEV VM. I wonder if there are BIOS settings on the hosts about going into standby?
Well now this is very strange. That P2V we started up Friday afternoon crossed into hour 52 a few minutes ago. But in the past 3 1/2 hours, it hasn't used one more byte of disk space in the NFS Export service. But then after firing up an strace just now, it started using space again. What's up with that?
- Greg
This time after roughly 7 1/2 hours, the exact amount of disk space used in the NFS Export domain. And then as soon as I run df a couple of times, now the disk space usage starts to climb again. One other symptom to note - that 3rd df took nearly 2 minutes to respond. See the date commands before and after. The first 2 df commands responded immediately. After the 2nd df command, I looked at virt-p2v-server with strace again. I don't think my RHEV hosts are going into standby. If they were, then why would my df and strace commands to a guest VM on one of the hosts respond immediately? I think the Storagetek NFS server goes offline and somehow when I either do a df command or strace against virt-p2v-server, it goes back online again. We'll know more when we try this same thing with a different NFS server hopefully after the sun comes up today. [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# date Sun Dec 4 19:49:06 CST 2011 [root@Fedora16-64P2V log]# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 49058108 3944084 42622020 9% / devtmpfs 2014608 0 2014608 0% /dev tmpfs 2024924 444 2024480 1% /dev/shm tmpfs 2024924 48480 1976444 3% /run /dev/mapper/vg_fedora1664p2v-lv_root 49058108 3944084 42622020 9% / tmpfs 2024924 48480 1976444 3% /run tmpfs 2024924 0 2024924 0% /sys/fs/cgroup tmpfs 2024924 0 2024924 0% /media /dev/vda2 495844 82656 387588 18% /boot 175.10.0.45:/EXPORTS/ 1055113408 93212416 961900992 9% /tmp/UBKmK8QbfM [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# [root@Fedora16-64P2V log]# date Mon Dec 5 03:10:49 CST 2011 [root@Fedora16-64P2V log]# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 49058108 3944144 42621960 9% / devtmpfs 2014608 0 2014608 0% /dev tmpfs 2024924 444 2024480 1% /dev/shm tmpfs 2024924 48476 1976448 3% /run /dev/mapper/vg_fedora1664p2v-lv_root 49058108 3944144 42621960 9% / tmpfs 2024924 48476 1976448 3% /run tmpfs 2024924 0 2024924 0% /sys/fs/cgroup tmpfs 2024924 0 2024924 0% /media /dev/vda2 495844 82656 387588 18% /boot 175.10.0.45:/EXPORTS/ 1055113408 93212416 961900992 9% /tmp/UBKmK8QbfM [root@Fedora16-64P2V log]# date Mon Dec 5 03:12:37 CST 2011 [root@Fedora16-64P2V log]# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 49058108 3944144 42621960 9% / devtmpfs 2014608 0 2014608 0% /dev tmpfs 2024924 444 2024480 1% /dev/shm tmpfs 2024924 48476 1976448 3% /run /dev/mapper/vg_fedora1664p2v-lv_root 49058108 3944144 42621960 9% / tmpfs 2024924 48476 1976448 3% /run tmpfs 2024924 0 2024924 0% /sys/fs/cgroup tmpfs 2024924 0 2024924 0% /media /dev/vda2 495844 82656 387588 18% /boot 175.10.0.45:/EXPORTS/ 1055113408 93266752 961846656 9% /tmp/UBKmK8QbfM [root@Fedora16-64P2V log]# date Mon Dec 5 03:14:18 CST 2011 [root@Fedora16-64P2V log]# - Greg
Very strange behavior. Fredy killed the P2V at the source server console about 3 hours ago. After roughly 67 hours, the bar graph on the console was at 60 percent. I just checked and noticed virt-p2v-server was still running on the conversion server. So I did an strace in a different window. Tons of output immediately poured out and virt-p2v-server ended. Below is what I still have in the putty window's buffer. I'll bet it was stalled writing to the NFS Export share. Why doing an strace makes it come alive again is a mystery. Or a miracle, depending on your point of view I guess. - Greg 14:28:44.438657 read(0, "\0\1\0\0\0\0\0\0\177\0\3\0z\0314\0Z;;\0;\4n\0K\6Z\0v\37C\0"..., 8192) = 8192 <0.000033> 14:28:44.439021 read(0, "\0\0\0002\0\0\0\22\0\0\0\10\0\0\0\1\0\0\0\0\0\0\0\27N\0\1\313\324i9\377"..., 8192) = 8192 <0.000033> 14:28:44.439357 read(0, "\364\265\202\377\367\271\205\377\370\262u\377\371\254f\377\373\252\\\377\374\263b\377\365\275o\377z)\31\337"..., 8192) = 8192 <0.000032> 14:28:44.439686 read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10ww"..., 8192) = 8192 <0.000032> 14:28:44.440057 read(0, "", 8192) = 0 <0.000028> 14:28:44.440390 mremap(0x7fe85e187000, 4198400, 8294400, MREMAP_MAYMOVE) = 0x7fe85cd9b000 <0.000931> 14:28:44.458613 write(1, "ERROR Error receiving data: \n\n", 30) = -1 EPIPE (Broken pipe) <0.000139> 14:28:44.459051 --- {si_signo=SIGPIPE, si_code=SI_USER, si_pid=23339, si_uid=0} (Broken pipe) --- 14:28:44.459565 sendto(3, "<11>Dec 5 14:28:44 virt-v2v[233"..., 67, MSG_NOSIGNAL, NULL, 0) = 67 <0.001088> 14:28:44.460994 write(2, "virt-v2v: Error receiving data: "..., 34) = 34 <0.000064> 14:28:44.461376 rt_sigaction(SIG_0, NULL, {0x34772bbf32, ~[RT_1 RT_2 RT_3 RT_4 RT_5 RT_6 RT_7 RT_8 RT_9 RT_10 RT_11 RT_12 RT_13 RT_14 RT_15 RT_16 RT_17 RT_18 RT_19 RT_20 RT_21 RT_22 RT_23 RT_24 RT_25 RT_26 RT_27 RT_28 RT_29 RT_30 RT_31], 0}, 8) = -1 EINVAL (Invalid argument) <0.000029> 14:28:44.461746 rt_sigaction(SIGHUP, NULL, {SIG_DFL, [], SA_RESTORER, 0x347620f4f0}, 8) = 0 <0.000028> 14:28:44.462108 rt_sigaction(SIGINT, NULL, {SIG_DFL, [], SA_RESTORER, 0x347620f4f0}, 8) = 0 <0.000028> 14:28:44.462420 rt_sigaction(SIGQUIT, NULL, {SIG_DFL, [], SA_RESTORER, 0x347620f4f0}, 8) = 0 <0.000028> 14:28:44.462728 rt_sigaction(SIGILL, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.463081 rt_sigaction(SIGTRAP, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000026> 14:28:44.463389 rt_sigaction(SIGABRT, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000026> 14:28:44.463695 rt_sigaction(SIGBUS, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000026> 14:28:44.464062 rt_sigaction(SIGFPE, NULL, {SIG_IGN, [FPE], SA_RESTORER|SA_RESTART, 0x3475e36300}, 8) = 0 <0.000025> 14:28:44.464374 rt_sigaction(SIGKILL, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000026> 14:28:44.464680 rt_sigaction(SIGUSR1, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.465027 rt_sigaction(SIGSEGV, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000084> 14:28:44.465483 rt_sigaction(SIGUSR2, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000036> 14:28:44.465865 rt_sigaction(SIGPIPE, NULL, {SIG_IGN, [], SA_RESTORER, 0x347620f4f0}, 8) = 0 <0.000028> 14:28:44.466275 rt_sigaction(SIGALRM, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000027> 14:28:44.466648 rt_sigaction(SIGTERM, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000027> 14:28:44.467042 rt_sigaction(SIGSTKFLT, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000033> 14:28:44.467401 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000026> 14:28:44.467712 rt_sigaction(SIGCONT, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.468088 rt_sigaction(SIGSTOP, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.468397 rt_sigaction(SIGTSTP, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.468703 rt_sigaction(SIGTTIN, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.469059 rt_sigaction(SIGTTOU, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.469368 rt_sigaction(SIGURG, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000026> 14:28:44.469674 rt_sigaction(SIGXCPU, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.470022 rt_sigaction(SIGXFSZ, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.470333 rt_sigaction(SIGVTALRM, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.470637 rt_sigaction(SIGPROF, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000026> 14:28:44.470975 rt_sigaction(SIGWINCH, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000026> 14:28:44.471293 rt_sigaction(SIGIO, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.471599 rt_sigaction(SIGPWR, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.471904 rt_sigaction(SIGSYS, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000026> 14:28:44.472272 rt_sigaction(SIGRT_2, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.472580 rt_sigaction(SIGRT_3, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000025> 14:28:44.472887 rt_sigaction(SIGRT_4, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000095> 14:28:44.473287 rt_sigaction(SIGRT_5, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000079> 14:28:44.473594 rt_sigaction(SIGRT_6, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000076> 14:28:44.473883 rt_sigaction(SIGRT_7, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000079> 14:28:44.474207 rt_sigaction(SIGRT_8, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000055> 14:28:44.474480 rt_sigaction(SIGRT_9, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000053> 14:28:44.474753 rt_sigaction(SIGRT_10, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000053> 14:28:44.475071 rt_sigaction(SIGRT_11, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000054> 14:28:44.475350 rt_sigaction(SIGRT_12, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000053> 14:28:44.475621 rt_sigaction(SIGRT_13, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000056> 14:28:44.475894 rt_sigaction(SIGRT_14, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000079> 14:28:44.476209 rt_sigaction(SIGRT_15, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000055> 14:28:44.476481 rt_sigaction(SIGRT_16, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000055> 14:28:44.476754 rt_sigaction(SIGRT_17, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000054> 14:28:44.477071 rt_sigaction(SIGRT_18, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000054> 14:28:44.477350 rt_sigaction(SIGRT_19, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000054> 14:28:44.477621 rt_sigaction(SIGRT_20, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000055> 14:28:44.477898 rt_sigaction(SIGRT_21, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000046> 14:28:44.478188 rt_sigaction(SIGRT_22, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000078> 14:28:44.478494 rt_sigaction(SIGRT_23, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000078> 14:28:44.478797 rt_sigaction(SIGRT_24, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000074> 14:28:44.479286 rt_sigaction(SIGRT_25, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000076> 14:28:44.479595 rt_sigaction(SIGRT_26, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000078> 14:28:44.479900 rt_sigaction(SIGRT_27, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000082> 14:28:44.480220 rt_sigaction(SIGRT_28, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000078> 14:28:44.480523 rt_sigaction(SIGRT_29, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000078> 14:28:44.480827 rt_sigaction(SIGRT_30, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000114> 14:28:44.481180 rt_sigaction(SIGRT_31, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000076> 14:28:44.481484 rt_sigaction(SIGRT_32, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000078> 14:28:44.481788 rt_sigaction(SIGABRT, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000076> 14:28:44.482118 rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000076> 14:28:44.482418 rt_sigaction(SIGIO, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000076> 14:28:44.482720 rt_sigaction(SIGSYS, NULL, {SIG_DFL, [], 0}, 8) = 0 <0.000075> 14:28:44.496140 lseek(4, 0, SEEK_CUR) = 34 <0.000018> 14:28:44.498511 sendto(3, "<12>Dec 5 14:28:44 virt-v2v[233"..., 118, MSG_NOSIGNAL, NULL, 0) = 118 <0.000135> 14:28:44.499006 sendto(3, "<13>Dec 5 14:28:44 virt-v2v[233"..., 56, MSG_NOSIGNAL, NULL, 0) = 56 <0.000143> 14:28:44.499587 close(4) = 0 <0.000017> 14:28:44.500350 close(5) = 0 <0.000885> 14:28:44.502847 rt_sigprocmask(SIG_BLOCK, [QUIT], [], 8) = 0 <0.000085> 14:28:44.503111 rt_sigaction(SIGQUIT, {0x3477298660, [], SA_RESTORER, 0x347620f4f0}, {SIG_DFL, [], SA_RESTORER, 0x347620f4f0}, 8) = 0 <0.000015> 14:28:44.503313 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000015> 14:28:44.503487 rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0 <0.000015> 14:28:44.503680 rt_sigaction(SIGINT, {0x3477298660, [], SA_RESTORER, 0x347620f4f0}, {SIG_DFL, [], SA_RESTORER, 0x347620f4f0}, 8) = 0 <0.000015> 14:28:44.503877 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000020> 14:28:44.504122 getgroups(0, NULL) = 1 <0.000016> 14:28:44.504302 getgroups(1, [36]) = 1 <0.000014> 14:28:44.504483 open("/proc/sys/kernel/ngroups_max", O_RDONLY) = 4 <0.000032> 14:28:44.504680 read(4, "65536\n", 31) = 6 <0.000020> 14:28:44.504842 close(4) = 0 <0.000018> 14:28:44.505049 setgroups(1, [36]) = 0 <0.000018> 14:28:44.505220 setresgid(-1, 36, -1) = 0 <0.000019> 14:28:44.505383 getegid() = 36 <0.000014> 14:28:44.505554 setresuid(-1, 36, -1) = 0 <0.000019> 14:28:44.505717 geteuid() = 36 <0.000014> 14:28:44.505925 pipe([4, 5]) = 0 <0.000036> 14:28:44.506168 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe861b939d0) = 26552 <0.001974> 14:28:44.508338 close(5) = 0 <0.000014> 14:28:44.508455 rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x347620f4f0}, {0x3477298660, [], SA_RESTORER, 0x347620f4f0}, 8) = 0 <0.000014> 14:28:44.508601 rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x347620f4f0}, {0x3477298660, [], SA_RESTORER, 0x347620f4f0}, 8) = 0 <0.000014> 14:28:44.508746 wait4(26552, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 26552 <0.569418> 14:28:45.078506 --- {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=26552, si_status=0, si_utime=0, si_stime=39} (Child exited) --- 14:28:45.078779 rt_sigaction(SIGINT, {0x3477298660, [], SA_RESTORER, 0x347620f4f0}, NULL, 8) = 0 <0.000264> 14:28:45.079238 rt_sigaction(SIGQUIT, {0x3477298660, [], SA_RESTORER, 0x347620f4f0}, NULL, 8) = 0 <0.000094> 14:28:45.079510 read(4, "", 4) = 0 <0.000093> 14:28:45.079744 close(4) = 0 <0.000097> 14:28:45.080136 open("/proc/sys/kernel/ngroups_max", O_RDONLY) = 4 <0.000115> 14:28:45.080399 read(4, "65536\n", 31) = 6 <0.000094> 14:28:45.080633 close(4) = 0 <0.000091> 14:28:45.080886 setgroups(1, [36]) = -1 EPERM (Operation not permitted) <0.000105> 14:28:45.081263 setresgid(-1, 0, -1) = 0 <0.000095> 14:28:45.081491 getegid() = 0 <0.000088> 14:28:45.081732 setresuid(-1, 0, -1) = 0 <0.000102> 14:28:45.082031 geteuid() = 0 <0.000089> 14:28:45.082278 rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0 <0.000096> 14:28:45.082603 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x347620f4f0}, {0x3477298660, [], SA_RESTORER, 0x347620f4f0}, 8) = 0 <0.000097> 14:28:45.082877 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000138> 14:28:45.083215 rt_sigprocmask(SIG_BLOCK, [QUIT], [], 8) = 0 <0.000095> 14:28:45.083480 rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x347620f4f0}, {0x3477298660, [], SA_RESTORER, 0x347620f4f0}, 8) = 0 <0.000097> 14:28:45.083754 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000098> 14:28:45.084272 open("/dev/null", O_RDONLY) = 4 <0.000178> 14:28:45.084574 ioctl(4, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff884f1a98) = -1 ENOTTY (Inappropriate ioctl for device) <0.000076> 14:28:45.086381 lseek(4, 0, SEEK_CUR) = 0 <0.000028> 14:28:45.086570 fstat(4, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0 <0.000029> 14:28:45.086774 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 <0.000028> 14:28:45.087449 stat("/tmp/", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0 <0.000033> 14:28:45.087673 stat("/tmp/", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0 <0.000031> 14:28:45.088018 open("/tmp/w6utybl9HW", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW, 0600) = 5 <0.000490> 14:28:45.088683 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff884f1aa8) = -1 ENOTTY (Inappropriate ioctl for device) <0.000029> 14:28:45.088871 lseek(5, 0, SEEK_CUR) = 0 <0.000029> 14:28:45.089110 fstat(5, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 <0.000029> 14:28:45.089311 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 <0.000027> 14:28:45.089497 chmod("/tmp/w6utybl9HW", 0600) = 0 <0.000044> 14:28:45.089775 fstat(5, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 <0.000029> 14:28:45.090044 stat("/tmp/w6utybl9HW", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 <0.000033> 14:28:45.090275 stat("/tmp/w6utybl9HW", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 <0.000032> 14:28:45.090567 stat("/tmp/w6utybl9HW", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 <0.000033> 14:28:45.090786 lstat("/tmp/w6utybl9HW", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 <0.000032> 14:28:45.091027 unlink("/tmp/w6utybl9HW") = 0 <0.000049> 14:28:45.091234 fstat(5, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 <0.000028> 14:28:45.091618 close(4) = 0 <0.000032> 14:28:45.091815 pipe([4, 7]) = 0 <0.000066> 14:28:45.092046 ioctl(4, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff884f1c68) = -1 EINVAL (Invalid argument) <0.000029> 14:28:45.092233 lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) <0.000029> 14:28:45.092429 ioctl(7, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff884f1c68) = -1 EINVAL (Invalid argument) <0.000028> 14:28:45.092612 lseek(7, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) <0.000027> 14:28:45.092791 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 <0.000027> 14:28:45.092997 fcntl(7, F_SETFD, FD_CLOEXEC) = 0 <0.000028> 14:28:45.093198 close(5) = 0 <0.000051> 14:28:45.093404 pipe([5, 8]) = 0 <0.000036> 14:28:45.093598 ioctl(5, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff884f1c68) = -1 EINVAL (Invalid argument) <0.000029> 14:28:45.093781 lseek(5, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) <0.000027> 14:28:45.094014 ioctl(8, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff884f1c68) = -1 EINVAL (Invalid argument) <0.000029> 14:28:45.094206 lseek(8, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) <0.000027> 14:28:45.094390 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 <0.000028> 14:28:45.094567 fcntl(8, F_SETFD, FD_CLOEXEC) = 0 <0.000028> 14:28:45.094798 pipe([9, 10]) = 0 <0.000064> 14:28:45.095037 ioctl(9, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff884f1c68) = -1 EINVAL (Invalid argument) <0.000029> 14:28:45.095225 lseek(9, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) <0.000027> 14:28:45.095413 ioctl(10, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff884f1c68) = -1 EINVAL (Invalid argument) <0.000028> 14:28:45.095597 lseek(10, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) <0.000027> 14:28:45.095777 fcntl(9, F_SETFD, FD_CLOEXEC) = 0 <0.000028> 14:28:45.096019 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 <0.000028> 14:28:45.096229 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe861b939d0) = 26553 <0.001945> 14:28:45.098477 close(10) = 0 <0.000016> 14:28:45.100108 read(9, "", 8192) = 0 <0.018531> 14:28:45.118890 close(9) = 0 <0.000169> 14:28:45.119270 close(4) = 0 <0.000085> 14:28:45.119528 close(8) = 0 <0.000086> 14:28:45.119847 wait4(26553, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 26553 <0.094938> 14:28:45.214994 --- {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=26553, si_status=0, si_utime=0, si_stime=0} (Child exited) --- 14:28:45.215204 close(7) = 0 <0.000098> 14:28:45.215489 rmdir("/tmp/UBKmK8QbfM") = 0 <0.000312> 14:28:45.216018 close(5) = 0 <0.000098> 14:28:45.232037 exit_group(32) = ? Process 23339 detached [root@Fedora16-64P2V ~]#
Trying a P2V again, this time using a physical Fedora system for the NFS RHEV Export domain instead of the Storagetek NFS server. Everything else is the same. Same source server, same conversion server, same v2v and p2v versions for now. The only variable different is the RHEV Export NFS server. Nothing is ever simple. For our new F16 NFS server, with the new systemd, the startup rules have changed. For anyone else who needs to use an F16 system as an NFS server for RHEV exports or maybe ISOs, here are some notes. YMMV, but this worked for me. Thanks to several forums for the raw materials that went into the notes below. - Greg **************************************************** First things first. We need a usable text editor. yum install nano Now we should change selinux to "permissive". [root@IT etc]# cd /etc/selinux [root@IT selinux]# ls config restorecond.conf restorecond_user.conf semanage.conf targeted [root@IT selinux]# nano config [root@IT selinux]# more config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - SELinux is fully disabled. SELINUX=permissive # SELINUXTYPE= type of policy in use. Possible values are: # targeted - Only targeted network daemons are protected. # strict - Full SELinux protection. SELINUXTYPE=targeted [root@IT selinux]# Now install the packages we need for NFS. yum install nfs-utils (This should pull in all the dependencies including rpcbind.) yum install system-config-nfs yum install system-config-nfs-docs Set them to start at boot time. And open up all the firewall rules. systemctl enable rpcbind.service systemctl enable nfs-server.service systemctl disable iptables.service Start everything up for now. systemctl stop iptables.service systemsctl start rpcbind.service systemctl start nfs-server.service And now we can set up our NFS share and edit /etc/exports. Where is the best place to put the NFS share we want? [root@IT /]# df -h Filesystem Size Used Avail Use% Mounted on rootfs 50G 2.7G 47G 6% / devtmpfs 1.5G 0 1.5G 0% /dev tmpfs 1.5G 240K 1.5G 1% /dev/shm tmpfs 1.5G 40M 1.4G 3% /run /dev/mapper/vg_it-lv_root 50G 2.7G 47G 6% / tmpfs 1.5G 40M 1.4G 3% /run tmpfs 1.5G 0 1.5G 0% /sys/fs/cgroup tmpfs 1.5G 0 1.5G 0% /media /dev/sde3 485M 56M 404M 13% /boot /dev/mapper/vg_it-lv_home 212G 252M 201G 1% /home [root@IT /]# ls /home fredy lost+found [root@IT /]# mkdir /home/RHEVexports [root@IT /]# chown 36.36 /home/RHEVexports [root@IT /]# ls -al /home total 32 drwxr-xr-x. 5 root root 4096 Dec 5 20:12 . dr-xr-xr-x. 21 root root 4096 Dec 5 11:49 .. drwx------. 25 fredy fredy 4096 Dec 5 11:49 fredy drwx------. 2 root root 16384 Nov 29 06:32 lost+found drwxr-xr-x. 2 36 36 4096 Dec 5 20:12 RHEVexports [root@IT etc]# nano /etc/exports [root@IT etc]# more /etc/exports /home/RHEVexports *(rw,sync,nohide,no_root_squash) [root@IT etc]# exportfs -av exporting *:/home/RHEVexports [root@IT etc]# We decided /home/RHEVexports seems like a reasonable location. And now it's mountable from a different system: [root@Fedora16-64P2V log]# mkdir /mnt/stuff [root@Fedora16-64P2V log]# mount -t nfs 175.10.0.95:/home/RHEVexports /mnt/stuff [root@Fedora16-64P2V log]# cd /mnt/stuff [root@Fedora16-64P2V stuff]# touch a.a [root@Fedora16-64P2V stuff]# rm a.a rm: remove regular empty file `a.a'? y [root@Fedora16-64P2V stuff]# cd / [root@Fedora16-64P2V /]# umount /mnt/stuff [root@Fedora16-64P2V /]# Should be ready to use as the RHEV-M export domain now.
participants (2)
-
Fredy Hernández -
Greg Scott