guestfish runs w/ a nested guest
# Ref: http://libguestfs.org/guestfs-performance.1.html Run the below command: $ time guestfish -a /dev/null run NOTE: Discard the first few results, to get a hot cache. (Thanks Rich.) 1/ L0. with L1 running. ---------------------------------------------------------------------- $ for i in {1..10}; do time guestfish -a /dev/null run; done real 0m28.277s user 0m11.028s sys 0m0.706s real 0m2.753s user 0m0.054s sys 0m0.049s real 0m2.771s user 0m0.063s sys 0m0.044s real 0m2.774s user 0m0.053s sys 0m0.044s real 0m2.772s user 0m0.054s sys 0m0.049s real 0m2.715s user 0m0.051s sys 0m0.045s real 0m2.753s user 0m0.064s sys 0m0.037s real 0m2.709s user 0m0.059s sys 0m0.044s real 0m2.782s user 0m0.055s sys 0m0.045s real 0m2.775s user 0m0.054s sys 0m0.049s ---------------------------------------------------------------------- => Remove the appliance and re-run: ---------------------------------------------------------------------- $ rm -rf /var/tmp/guestfs-0 $ for in in {1..2}; do time guestfish -a /dev/null run; done real 0m14.623s user 0m10.172s sys 0m0.520s real 0m2.790s user 0m0.048s sys 0m0.046s ---------------------------------------------------------------------- 2. L1 w/ nesting, VMCS shadowing enabled. NOTE: *NO* L2 guests are running while this test is being performed. ---------------------------------------------------------------------- $ for i in {1..10}; do time guestfish -a /dev/null run; done | tee guestfish-timings-L1.txt real 0m55.182s user 0m14.747s sys 0m8.501s real 0m8.754s user 0m0.054s sys 0m0.494s real 0m9.576s user 0m0.056s sys 0m0.490s real 0m8.793s user 0m0.056s sys 0m0.498s real 0m8.767s user 0m0.053s sys 0m0.492s real 0m8.715s user 0m0.049s sys 0m0.502s real 0m8.814s user 0m0.051s sys 0m0.503s real 0m8.765s user 0m0.046s sys 0m0.516s real 0m8.783s user 0m0.050s sys 0m0.502s real 0m8.739s user 0m0.054s sys 0m0.496s ---------------------------------------------------------------------- => Next, is to try: L1 w/o nesting, VMCS shadowing disabled. ---------------------------------------------------------------------- # On L0 $ vi /etc/modprobe.d/dist.conf - s/options kvm-intel nested=y/options kvm-intel nested=n/ # For the test purpose, remove the XML fragment (which adds VMX # extensions) for L1 CPU $ virsh edit regular-guest - Remove the below fragment, save it: <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> <feature policy='require' name='vmx'/> </cpu> # Reboot $ init 6 # Check for the nested parameter on L0 $ cat /sys/module/kvm_intel/parameters/nested N # Remove any old appliances $ rm -rf /var/tmp/.guestfs-0 # Run: $ for i in {1..10}; do time guestfish -a /dev/null run; done | tee \ guestfish-timings-L1-no-nested.txt ---------------------------------------------------------------------- -- /kashyap
=> Next, is to try: L1 w/o nesting, VMCS shadowing disabled. ----------------------------------------------------------------------
# On L0 $ vi /etc/modprobe.d/dist.conf - s/options kvm-intel nested=y/options kvm-intel nested=n/
# For the test purpose, remove the XML fragment (which adds VMX # extensions) for L1 CPU $ virsh edit regular-guest - Remove the below fragment, save it: <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> <feature policy='require' name='vmx'/> </cpu>
# Reboot $ init 6
# Check for the nested parameter on L0 $ cat /sys/module/kvm_intel/parameters/nested N
# Remove any old appliances $ rm -rf /var/tmp/.guestfs-0
# Run: $ for i in {1..10}; do time guestfish -a /dev/null run; done | tee \ guestfish-timings-L1-no-nested.txt ----------------------------------------------------------------------
Results for the above case -- no nesting, guestfish run in 1st level guest: ---------------------------------------------------------------------- $ for i in {1..10}; do time guestfish -a /dev/null run; done | tee guestfish-timings-L1-no-nesting.txt real 1m10.899s user 0m13.558s sys 0m8.167s real 0m29.899s user 0m0.066s sys 0m0.522s real 0m25.310s user 0m0.056s sys 0m0.516s real 0m35.267s user 0m0.053s sys 0m0.515s real 0m24.872s user 0m0.051s sys 0m0.545s real 0m24.827s user 0m0.044s sys 0m0.538s real 0m41.053s user 0m0.060s sys 0m0.531s real 0m29.107s user 0m0.049s sys 0m0.534s real 0m30.196s user 0m0.051s sys 0m0.532s real 0m52.815s user 0m0.052s sys 0m0.520s ---------------------------------------------------------------------- To summarize: L0 (bare-metal): - guestfish run: ~ 3 seconds L1 (guest): - w/o nesting, guestfish run: ~ 52 seconds - w/ nesting, guestfish run: ~ 8 seconds (Almost 6 1/2 times speed improvement) For Reference: -------------- 1/ Level-1 guest qemu-kvm command-line, indicating that I *did* remove <cpu> fragment gracefully, i.e. no nesting. =================================== $ ps -ef | grep -i qemu qemu 2367 1 86 08:29 ? 00:00:03 /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 =================================== 2/ Level-1 guest qemu-kvm command-line *with* nesting: =================================== $ ps -ef | grep -i qemu qemu 4962 1 21 15:41 ? 00:00:41 /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 =================================== The setup details are here - https://raw.github.com/kashyapc/nvmx-haswell/master/SETUP-nVMX.rst Rich, if you have further thoughts, please let me know. Thanks. -- /kashyap
To summarize:
L0 (bare-metal): - guestfish run: ~ 3 seconds
L1 (guest): - w/o nesting, guestfish run: ~ 52 seconds - w/ nesting, guestfish run: ~ 8 seconds (Almost 6 1/2 times speed improvement)
To clarify, by "w/ nesting" what I meant is, L1's qemu-kvm cli will have: -cpu Core2Duo,+vmx # or whatever the model is And, w/o nesting": No -cpu parameter in L1's qemu-kvm cli. -- /kashyap
On Tue, May 14, 2013 at 06:14:11PM +0530, Kashyap Chamarthy wrote:
To summarize:
L0 (bare-metal): - guestfish run: ~ 3 seconds
L1 (guest): - w/o nesting, guestfish run: ~ 52 seconds - w/ nesting, guestfish run: ~ 8 seconds (Almost 6 1/2 times speed improvement)
Or to put it another way, nested hardware virtualization is over twice as slow as ordinary hardware virtualization :-) However it's still a lot better than TCG, so this is good. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 05/16/2013 09:12 PM, Richard W.M. Jones wrote:
On Tue, May 14, 2013 at 06:14:11PM +0530, Kashyap Chamarthy wrote:
To summarize:
L0 (bare-metal): - guestfish run: ~ 3 seconds
L1 (guest): - w/o nesting, guestfish run: ~ 52 seconds - w/ nesting, guestfish run: ~ 8 seconds (Almost 6 1/2 times speed improvement)
Or to put it another way, nested hardware virtualization is over twice as slow as ordinary hardware virtualization :-)
:) But doesn't it look optimistic: given all the I/O, MMU, related on-going research, it might only get better ?
However it's still a lot better than TCG, so this is good.
Yes, I have on my TODO to write more coherent details after some further tests. Let me see if I could muster enough discipline to do it. Now that I mentioned here, it might put little more pressure on me. As an aside, when you get a moment, can you please enlighten a bit more on how often we (in Fedora, elsewhere) use TCG ? Meantime, I'm reading - http://git.qemu.org/?p=qemu.git;a=blob_plain;f=tcg/README;hb=HEAD Thanks.
Rich.
-- /kashyap
On Thu, May 16, 2013 at 10:03:57PM +0530, Kashyap Chamarthy wrote:
As an aside, when you get a moment, can you please enlighten a bit more on how often we (in Fedora, elsewhere) use TCG ?
All the time: - When building libguestfs (in Koji). - Whenever using libguestfs in a VM, eg. for testing, development. - Commonly when deploying virt-v2v. - In the cloud case (eg. running libguestfs in an Amazon EC2 instance). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 05/16/2013 10:30 PM, Richard W.M. Jones wrote:
On Thu, May 16, 2013 at 10:03:57PM +0530, Kashyap Chamarthy wrote:
As an aside, when you get a moment, can you please enlighten a bit more on how often we (in Fedora, elsewhere) use TCG ?
All the time:
- When building libguestfs (in Koji).
- Whenever using libguestfs in a VM, eg. for testing, development.
- Commonly when deploying virt-v2v.
- In the cloud case (eg. running libguestfs in an Amazon EC2 instance).
Thanks for this detail, Rich. -- /kashyap
participants (2)
-
Kashyap Chamarthy -
Richard W.M. Jones