7:11 a.m.
# Ref: http://libguestfs.org/guestfs-performance.1.html
Run the below command:
$ time guestfish -a /dev/null run
NOTE: Discard the first few results, to get a hot cache. (Thanks Rich.)
1/ L0. with L1 running.
----------------------------------------------------------------------
$ for i in {1..10}; do time guestfish -a /dev/null run; done
real 0m28.277s
user 0m11.028s
sys 0m0.706s
real 0m2.753s
user 0m0.054s
sys 0m0.049s
real 0m2.771s
user 0m0.063s
sys 0m0.044s
real 0m2.774s
user 0m0.053s
sys 0m0.044s
real 0m2.772s
user 0m0.054s
sys 0m0.049s
real 0m2.715s
user 0m0.051s
sys 0m0.045s
real 0m2.753s
user 0m0.064s
sys 0m0.037s
real 0m2.709s
user 0m0.059s
sys 0m0.044s
real 0m2.782s
user 0m0.055s
sys 0m0.045s
real 0m2.775s
user 0m0.054s
sys 0m0.049s
----------------------------------------------------------------------
=> Remove the appliance and re-run:
----------------------------------------------------------------------
$ rm -rf /var/tmp/guestfs-0
$ for in in {1..2}; do time guestfish -a /dev/null run; done
real 0m14.623s
user 0m10.172s
sys 0m0.520s
real 0m2.790s
user 0m0.048s
sys 0m0.046s
----------------------------------------------------------------------
2. L1 w/ nesting, VMCS shadowing enabled. NOTE: *NO* L2 guests are running
while this test is being performed.
----------------------------------------------------------------------
$ for i in {1..10}; do time guestfish -a /dev/null run; done | tee
guestfish-timings-L1.txt
real 0m55.182s
user 0m14.747s
sys 0m8.501s
real 0m8.754s
user 0m0.054s
sys 0m0.494s
real 0m9.576s
user 0m0.056s
sys 0m0.490s
real 0m8.793s
user 0m0.056s
sys 0m0.498s
real 0m8.767s
user 0m0.053s
sys 0m0.492s
real 0m8.715s
user 0m0.049s
sys 0m0.502s
real 0m8.814s
user 0m0.051s
sys 0m0.503s
real 0m8.765s
user 0m0.046s
sys 0m0.516s
real 0m8.783s
user 0m0.050s
sys 0m0.502s
real 0m8.739s
user 0m0.054s
sys 0m0.496s
----------------------------------------------------------------------
=> Next, is to try: L1 w/o nesting, VMCS shadowing disabled.
----------------------------------------------------------------------
# On L0
$ vi /etc/modprobe.d/dist.conf
- s/options kvm-intel nested=y/options kvm-intel nested=n/
# For the test purpose, remove the XML fragment (which adds VMX
# extensions) for L1 CPU
$ virsh edit regular-guest
- Remove the below fragment, save it:
<cpu mode='custom' match='exact'>
<model fallback='allow'>Haswell</model>
<feature policy='require' name='vmx'/>
</cpu>
# Reboot
$ init 6
# Check for the nested parameter on L0
$ cat /sys/module/kvm_intel/parameters/nested
N
# Remove any old appliances
$ rm -rf /var/tmp/.guestfs-0
# Run:
$ for i in {1..10}; do time guestfish -a /dev/null run; done | tee \
guestfish-timings-L1-no-nested.txt
----------------------------------------------------------------------
--
/kashyap
7:44 a.m.
=> Next, is to try: L1 w/o nesting, VMCS shadowing disabled.
----------------------------------------------------------------------
# On L0
$ vi /etc/modprobe.d/dist.conf
- s/options kvm-intel nested=y/options kvm-intel nested=n/
# For the test purpose, remove the XML fragment (which adds VMX
# extensions) for L1 CPU
$ virsh edit regular-guest
- Remove the below fragment, save it:
<cpu mode='custom' match='exact'>
<model fallback='allow'>Haswell</model>
<feature policy='require' name='vmx'/>
</cpu>
# Reboot
$ init 6
# Check for the nested parameter on L0
$ cat /sys/module/kvm_intel/parameters/nested
N
# Remove any old appliances
$ rm -rf /var/tmp/.guestfs-0
# Run:
$ for i in {1..10}; do time guestfish -a /dev/null run; done | tee \
guestfish-timings-L1-no-nested.txt
----------------------------------------------------------------------
Results for the above case -- no nesting, guestfish run in 1st level guest:
----------------------------------------------------------------------
$ for i in {1..10}; do time guestfish -a /dev/null run; done | tee
guestfish-timings-L1-no-nesting.txt
real 1m10.899s
user 0m13.558s
sys 0m8.167s
real 0m29.899s
user 0m0.066s
sys 0m0.522s
real 0m25.310s
user 0m0.056s
sys 0m0.516s
real 0m35.267s
user 0m0.053s
sys 0m0.515s
real 0m24.872s
user 0m0.051s
sys 0m0.545s
real 0m24.827s
user 0m0.044s
sys 0m0.538s
real 0m41.053s
user 0m0.060s
sys 0m0.531s
real 0m29.107s
user 0m0.049s
sys 0m0.534s
real 0m30.196s
user 0m0.051s
sys 0m0.532s
real 0m52.815s
user 0m0.052s
sys 0m0.520s
----------------------------------------------------------------------
To summarize:
L0 (bare-metal):
- guestfish run: ~ 3 seconds
L1 (guest):
- w/o nesting, guestfish run: ~ 52 seconds
- w/ nesting, guestfish run: ~ 8 seconds (Almost 6 1/2 times speed improvement)
For Reference:
--------------
1/ Level-1 guest qemu-kvm command-line, indicating that I *did* remove <cpu>
fragment
gracefully, i.e. no nesting.
===================================
$ ps -ef | grep -i qemu
qemu 2367 1 86 08:29 ? 00:00:03 /usr/bin/qemu-system-x86_64 -machine
accel=kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -m 6144 -smp
4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device
usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
===================================
2/ Level-1 guest qemu-kvm command-line *with* nesting:
===================================
$ ps -ef | grep -i qemu
qemu 4962 1 21 15:41 ? 00:00:41 /usr/bin/qemu-system-x86_64 -machine
accel=kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu
Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid
4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device
usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
===================================
The setup details are here -
https://raw.github.com/kashyapc/nvmx-haswell/master/SETUP-nVMX.rst
Rich, if you have further thoughts, please let me know.
Thanks.
--
/kashyap
7:55 a.m.
To summarize:
L0 (bare-metal):
- guestfish run: ~ 3 seconds
L1 (guest):
- w/o nesting, guestfish run: ~ 52 seconds
- w/ nesting, guestfish run: ~ 8 seconds (Almost 6 1/2 times speed improvement)
To clarify, by "w/ nesting" what I meant is, L1's qemu-kvm cli will have:
-cpu Core2Duo,+vmx # or whatever the model is
And, w/o nesting": No -cpu parameter in L1's qemu-kvm cli.
--
/kashyap
10:42 a.m.
On Tue, May 14, 2013 at 06:14:11PM +0530, Kashyap Chamarthy wrote:
To summarize:
L0 (bare-metal):
- guestfish run: ~ 3 seconds
L1 (guest):
- w/o nesting, guestfish run: ~ 52 seconds
- w/ nesting, guestfish run: ~ 8 seconds (Almost 6 1/2 times speed improvement)
Or to put it another way, nested hardware virtualization is over twice
as slow as ordinary hardware virtualization :-)
However it's still a lot better than TCG, so this is good.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
11:33 a.m.
On 05/16/2013 09:12 PM, Richard W.M. Jones wrote:
> Rich.
--
/kashyap
On Tue, May 14, 2013 at 06:14:11PM +0530, Kashyap Chamarthy wrote:
> To summarize:
>
> L0 (bare-metal):
> - guestfish run: ~ 3 seconds
>
> L1 (guest):
> - w/o nesting, guestfish run: ~ 52 seconds
> - w/ nesting, guestfish run: ~ 8 seconds (Almost 6 1/2 times speed improvement)
Or to put it another way, nested hardware virtualization is over twice
as slow as ordinary hardware virtualization :-)
:)
But doesn't it look optimistic: given all the I/O, MMU, related on-going research, it
might only get better ?
However it's still a lot better than TCG, so this is good.
Yes, I have on my TODO to write more coherent details after some further tests. Let me
see
if I could muster enough discipline to do it. Now that I mentioned here, it might put
little more pressure on me.
As an aside, when you get a moment, can you please enlighten a bit more on how often we
(in Fedora, elsewhere) use TCG ?
Meantime, I'm reading -
http://git.qemu.org/?p=qemu.git;a=blob_plain;f=tcg/README;hb=HEAD
Thanks.
noon
On Thu, May 16, 2013 at 10:03:57PM +0530, Kashyap Chamarthy wrote:
As an aside, when you get a moment, can you please enlighten a bit
more on how often we
(in Fedora, elsewhere) use TCG ?
All the time:
- When building libguestfs (in Koji).
- Whenever using libguestfs in a VM, eg. for testing, development.
- Commonly when deploying virt-v2v.
- In the cloud case (eg. running libguestfs in an Amazon EC2 instance).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
10:17 p.m.
On 05/16/2013 10:30 PM, Richard W.M. Jones wrote:
On Thu, May 16, 2013 at 10:03:57PM +0530, Kashyap Chamarthy wrote:
> As an aside, when you get a moment, can you please enlighten a bit more on how often
we
> (in Fedora, elsewhere) use TCG ?
All the time:
- When building libguestfs (in Koji).
- Whenever using libguestfs in a VM, eg. for testing, development.
- Commonly when deploying virt-v2v.
- In the cloud case (eg. running libguestfs in an Amazon EC2 instance).
Thanks for this detail, Rich.
--
/kashyap
4207
days inactive
4210
days old
6 comments
2 participants
participants (2)
-
Kashyap Chamarthy -
Richard W.M. Jones