8:38 a.m.
Hello,
I have been looking at the "virt-df" libguestfs tool to get guest-level disk
used/free statistics - specifically with Qemu-KVM/Openstack. This works great for a few
Openstack instances, but when I begin to scale (even to ~30 instances/guests) the
performance really takes a hit. The time it takes for the command to complete seems to
scale linearly with the amount of guests/domains running on the hypervisor (note - I am
using "virt-df" for all guests, not specifying one at a time; although I've
tried that, too).
For ~30 guests, the "virt-df" command takes around 90 seconds to complete. We
are looking to support a scale of 3,000-30,000 guests disk used/free. It looks like this
won't be remotely possible using "virt-df".
Has anyone run into this same problem or something similar and can give me some
suggestions to find a workaround?
Thanks,
Dan Ryder
11:31 a.m.
On Wed, Sep 10, 2014 at 01:38:16PM +0000, Dan Ryder (daryder) wrote:
Hello,
I have been looking at the "virt-df" libguestfs tool to get
guest-level disk used/free statistics - specifically with
Qemu-KVM/Openstack. This works great for a few Openstack instances,
but when I begin to scale (even to ~30 instances/guests) the
performance really takes a hit. The time it takes for the command to
complete seems to scale linearly with the amount of guests/domains
running on the hypervisor (note - I am using "virt-df" for all
guests, not specifying one at a time; although I've tried that,
too).
For ~30 guests, the "virt-df" command takes around 90 seconds to
complete. We are looking to support a scale of 3,000-30,000 guests
disk used/free. It looks like this won't be remotely possible using
"virt-df".
With sufficient memory, non-nested, on hardware built in the last 3
years, you should get performance of about 1 second / guest
(pipelined). So the figure you give is about 3 times too high than
where it should be.
Just to get some basic things out of the way:
- What version of virt-df are you using and on what distro?
- Is this nested?
- What exact virt-df command(s) are you running?
- Are you using the -P option?
- How much free memory is on the system? virt-df runs multiple
threads in parallel, but the number to run is computed according to
the amount of free memory[1].
Also have a look at the guestfs-performance manual[2]. I'm especially
interested in the results of the baseline measurements on that page,
but the rest of the page should answer some of your questions too.
There's also an interesting Perl script you might try playing with.
Rich.
[1] https://github.com/libguestfs/libguestfs/blob/master/df/estimate-max-thre...
[2] http://libguestfs.org/guestfs-performance.1.html
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
2:35 p.m.
Hi Richard,
Thanks a lot for the response - very helpful to go through.
I'm using libguestfs 1.26.5 on an Ubuntu 14.04 OS, which is running as a baremetal
server. I was simply using "virt-df", but after looking into the "-P"
option a little more I have incorporated it. That greatly improves the performance
compared to the original (I checked the free memory available and it ranged from 2-3G,
which wouldn't have given me as many parallel threads running). Question about this -
is it safe to enforce a larger threadcount without checking available memory first?
I ran the baselines and got the following:
Starting the appliance = ~3.4s
Performing inspection of a guest = ~5s
I also looked at your blog posts - very interesting stuff. I played with setting
"LIBGUESTFS_ATTACH_METHOD=appliance", but didn't notice much difference
here. I'm testing on a Qemu-KVM/Openstack with 29 guests running. KVM has the default
client connections as 20, so I expected to see a little improvement when setting the above
env var. After setting, I changed "-P" to 29 instead of 20, but didn't see
any difference.
Any additional suggestions as the scale dramatically increases to > 3000 guests (This
will likely be on a system with much more available memory)? Ideally we would like to
gather guest disk used/free <5 minute intervals for all guests - do you think this is
possible using virt-df?
Thanks!
Dan Ryder
-----Original Message-----
From: Richard W.M. Jones [mailto:rjones@redhat.com]
Sent: Wednesday, September 10, 2014 12:32 PM
To: Dan Ryder (daryder)
Cc: libguestfs(a)redhat.com
Subject: Re: [Libguestfs] Scaling virt-df performance
On Wed, Sep 10, 2014 at 01:38:16PM +0000, Dan Ryder (daryder) wrote:
Hello,
I have been looking at the "virt-df" libguestfs tool to get
guest-level disk used/free statistics - specifically with
Qemu-KVM/Openstack. This works great for a few Openstack instances,
but when I begin to scale (even to ~30 instances/guests) the
performance really takes a hit. The time it takes for the command to
complete seems to scale linearly with the amount of guests/domains
running on the hypervisor (note - I am using "virt-df" for all guests,
not specifying one at a time; although I've tried that, too).
For ~30 guests, the "virt-df" command takes around 90 seconds to
complete. We are looking to support a scale of 3,000-30,000 guests
disk used/free. It looks like this won't be remotely possible using
"virt-df".
With sufficient memory, non-nested, on hardware built in the last 3 years, you should get
performance of about 1 second / guest (pipelined). So the figure you give is about 3
times too high than where it should be.
Just to get some basic things out of the way:
- What version of virt-df are you using and on what distro?
- Is this nested?
- What exact virt-df command(s) are you running?
- Are you using the -P option?
- How much free memory is on the system? virt-df runs multiple
threads in parallel, but the number to run is computed according to
the amount of free memory[1].
Also have a look at the guestfs-performance manual[2]. I'm especially interested in
the results of the baseline measurements on that page, but the rest of the page should
answer some of your questions too.
There's also an interesting Perl script you might try playing with.
Rich.
[1] https://github.com/libguestfs/libguestfs/blob/master/df/estimate-max-thre...
[2] http://libguestfs.org/guestfs-performance.1.html
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my
programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows
cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100
libraries supported.
http://fedoraproject.org/wiki/MinGW
3:25 p.m.
On Wed, Sep 10, 2014 at 07:35:03PM +0000, Dan Ryder (daryder) wrote:
Hi Richard,
Thanks a lot for the response - very helpful to go through.
I'm using libguestfs 1.26.5 on an Ubuntu 14.04 OS, which is
running
as a baremetal server. I was simply using "virt-df", but after
looking into the "-P" option a little more I have incorporated
it. That greatly improves the performance compared to the original
(I checked the free memory available and it ranged from 2-3G, which
wouldn't have given me as many parallel threads running). Question
about this - is it safe to enforce a larger threadcount without
checking available memory first?
Ah well, that's a question.
You can control the amount of memory given to the appliance by setting
LIBGUESTFS_MEMSIZE (integer number in megabytes, default 500).
However that doesn't mean that the appliance uses 500 MB -- it's just
malloc'd memory which is created on demand, and likely a very
short-running appliance like the one used by virt-df hardly touches
much memory.
A tool like perf, or even just getrusage(2), might help to understand
the memory requirements of each instance, and then you just multiply
by the number of threads (minus a tiny bit because qemu's code is
shared).
You could crank down LIBGUESTFS_MEMSIZE to the smallest size you think
you can get away with, and consequently run lots more threads.
I ran the baselines and got the following:
Starting the appliance = ~3.4s
Performing inspection of a guest = ~5s
Those are about right.
Notes:
(1) virt-df doesn't use inspection.
(2) Parallelized virt-df is, amortized, much faster. It can usually
clear guests at a rate of greater than 1 per second on my laptop.
I also looked at your blog posts - very interesting stuff. I played
with setting "LIBGUESTFS_ATTACH_METHOD=appliance", but didn't notice
much difference here.
On Ubuntu, that's the default! However even on Fedora, although this
avoids the overhead of libvirt, TBH any overhead is hardly noticable.
You can find the default setting by doing:
guestfish get-backend
(which on Ubuntu should print 'direct', which is the new name of
'appliance' and is exactly the same mode).
I'm testing on a Qemu-KVM/Openstack with 29 guests running. KVM
has
the default client connections as 20, so I expected to see a little
improvement when setting the above env var. After setting, I changed
"-P" to 29 instead of 20, but didn't see any difference.
I wasn't aware that KVM had a limit? libvirt has a client connection
limit of 20 (can be adjusted in /etc/libvirt/libvirtd.conf), but
you're probably not using libvirt.
Any additional suggestions as the scale dramatically increases to
>
3000 guests (This will likely be on a system with much more
available memory)? Ideally we would like to gather guest disk
used/free <5 minute intervals for all guests - do you think this is
possible using virt-df?
You'll have to run the memory numbers above. If you have lots of
memory, then maybe yes, but it won't be efficient.
You might consider using one appliance per lots of guests. With
virtio-scsi you can connect up to 254 disks per appliance (we could
actually increase this to effectively infinite with a smallish code
change). There are problems with this scenario, but it'll be a lot
faster. The problems are:
- Won't work if guests have filesystems or LVs with the same UUIDs
(happens when guests are cloned or run from templates).
- Also won't work if guests have LVs with the same name.
- Probably insecure if used with multi-tenancy, since in theory one
guest could "crack" the appliance and exploit other disks.
- virt-df doesn't support this so you'd have to write a bit of code
that calls g.statvfs and prints out the results. (See attachment)
If you were using libvirt, I could also suggest looking into
hot-plugging, although TBH hot-plug is almost as slow as running
separate appliances.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
3726
days inactive
3726
days old
3 comments
2 participants
participants (2)
-
Dan Ryder (daryder) -
Richard W.M. Jones