Re: libguestfs performance
On Tue, Dec 01, 2009 at 04:08:38PM +0100, Pierre Riteau wrote:
For a project of mine I would like to create VMs from tarballs of root filesystems. I would like to use libguestfs since this is exactly what the tar2vm.sh guestfish recipe does.
However, I need to extract tarballs that are in the hundreds of megabytes, but my tests showed that performance is very low, both with tgz-in and with guestmount (sorry I don't have the numbers any more, but it was something like more than one hour to extract a tarball containing a Debian Lenny installation).
Is there any way to drastically improve libguestfs' performance?
First of all, read: http://libguestfs.org/FAQ.html#slow Using tar-in/tgz-in to copy the tarball into the appliance is always going to be slow. I would think you should get better performance if you write the tarball into an ISO first, and attach that. Something like: mkisofs -JR -o cd.iso the_tarball.tar.gz guestfish -a guest.img -a cd.iso [...]
<fs> mkmountpoint /guest <fs> mkmountpoint /cd [...] <fs> mount /dev/sdb /cd [...] <fs> debug sh "tar xcf /cd/the_tarball.tar.gz /guest"
However we found experimentally that the fastest method of all for creating distributions was to prebuild an ext2/3/4 image, and 'dd' it over to the target partition, then resize it to the final (larger) size. This method is extremely quick, enabling us to build Fedora images in just a minute or two: http://rwmj.wordpress.com/2009/11/23/prebuilt-distributions-part-2/#content Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://et.redhat.com/~rjones/libguestfs/ See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html
On Mon, Dec 07, 2009 at 09:57:08AM +0000, Richard W.M. Jones wrote:
<fs> debug sh "tar xcf /cd/the_tarball.tar.gz /guest"
xcf? I mean zxf. Using debug sh is a hack. If you want to do it this way we should add a structured API call to pack and unpack local tarballs. However I would still recommend the 'dd+resize' approach, as it will always be much faster. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On 7 déc. 2009, at 10:57, Richard W.M. Jones wrote:
On Tue, Dec 01, 2009 at 04:08:38PM +0100, Pierre Riteau wrote:
For a project of mine I would like to create VMs from tarballs of root filesystems. I would like to use libguestfs since this is exactly what the tar2vm.sh guestfish recipe does.
However, I need to extract tarballs that are in the hundreds of megabytes, but my tests showed that performance is very low, both with tgz-in and with guestmount (sorry I don't have the numbers any more, but it was something like more than one hour to extract a tarball containing a Debian Lenny installation).
Is there any way to drastically improve libguestfs' performance?
First of all, read:
http://libguestfs.org/FAQ.html#slow
Using tar-in/tgz-in to copy the tarball into the appliance is always going to be slow. I would think you should get better performance if you write the tarball into an ISO first, and attach that. Something like:
mkisofs -JR -o cd.iso the_tarball.tar.gz
guestfish -a guest.img -a cd.iso [...]
<fs> mkmountpoint /guest <fs> mkmountpoint /cd [...] <fs> mount /dev/sdb /cd [...] <fs> debug sh "tar xcf /cd/the_tarball.tar.gz /guest"
I got it to work using the following sequence: $ guestfish -a guest.img -a cd.iso run sfdiskM /dev/sda "," mkfs ext3 /dev/sda1 mkmountpoint /guest mkmountpoint /cd mount /dev/sda1 /guest mount /dev/sdb /cd debug sh "tar -C /sysroot/guest -xzf /sysroot/cd/the_tarball.tar.gz" This is much better (on my test system it writes at about 15 MB/s). However, it is still far from what the hardware can do (more than 60 MB/s). Now the last problem is that I need to install a bootloader. Can I easily install grub in the appliance, or do I need to recreate a new one?
However we found experimentally that the fastest method of all for creating distributions was to prebuild an ext2/3/4 image, and 'dd' it over to the target partition, then resize it to the final (larger) size. This method is extremely quick, enabling us to build Fedora images in just a minute or two:
http://rwmj.wordpress.com/2009/11/23/prebuilt-distributions-part-2/#content
I've read your blog, this is interesting. I will see how I can leverage that. -- Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
On Mon, Dec 07, 2009 at 03:33:41PM +0100, Pierre Riteau wrote:
This is much better (on my test system it writes at about 15 MB/s). However, it is still far from what the hardware can do (more than 60 MB/s).
It'd be interesting to know whether the problem is with reading from the ISO image, or with writing to the disk, or with all the processing that tar and the virtual kernel is doing. Tricky to measure that though ...
Now the last problem is that I need to install a bootloader. Can I easily install grub in the appliance, or do I need to recreate a new one?
You should be able to issue the grub-install command: http://libguestfs.org/guestfish.1.html#grub_install IIRC the syntax is something like: grub-install /boot hd0
However we found experimentally that the fastest method of all for creating distributions was to prebuild an ext2/3/4 image, and 'dd' it over to the target partition, then resize it to the final (larger) size. This method is extremely quick, enabling us to build Fedora images in just a minute or two:
http://rwmj.wordpress.com/2009/11/23/prebuilt-distributions-part-2/#content
I've read your blog, this is interesting. I will see how I can leverage that.
I'm pretty certain this is the way to go. Note that the 'dd' command hasn't made it into a released version of libguestfs. It was added in this commit, just after 1.0.79 was released: http://git.annexia.org/?p=libguestfs.git;a=commit;h=22e531bc459309d9a871a845... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
Pierre Riteau wrote:
On 7 déc. 2009, at 10:57, Richard W.M. Jones wrote:
On Tue, Dec 01, 2009 at 04:08:38PM +0100, Pierre Riteau wrote:
For a project of mine I would like to create VMs from tarballs of root filesystems. I would like to use libguestfs since this is exactly what the tar2vm.sh guestfish recipe does.
However, I need to extract tarballs that are in the hundreds of megabytes, but my tests showed that performance is very low, both with tgz-in and with guestmount (sorry I don't have the numbers any more, but it was something like more than one hour to extract a tarball containing a Debian Lenny installation).
Is there any way to drastically improve libguestfs' performance?
First of all, read:
http://libguestfs.org/FAQ.html#slow
Using tar-in/tgz-in to copy the tarball into the appliance is always going to be slow. I would think you should get better performance if you write the tarball into an ISO first, and attach that. Something like:
mkisofs -JR -o cd.iso the_tarball.tar.gz
guestfish -a guest.img -a cd.iso [...]
<fs> mkmountpoint /guest <fs> mkmountpoint /cd [...] <fs> mount /dev/sdb /cd [...] <fs> debug sh "tar xcf /cd/the_tarball.tar.gz /guest"
I got it to work using the following sequence:
$ guestfish -a guest.img -a cd.iso run sfdiskM /dev/sda "," mkfs ext3 /dev/sda1 mkmountpoint /guest mkmountpoint /cd mount /dev/sda1 /guest mount /dev/sdb /cd debug sh "tar -C /sysroot/guest -xzf /sysroot/cd/the_tarball.tar.gz"
This is much better (on my test system it writes at about 15 MB/s). However, it is still far from what the hardware can do (more than 60 MB/s).
tar's default block size is historical, and probably too small. With a larger one you might narrow the gap.
participants (3)
-
Jim Meyering -
Pierre Riteau -
Richard W.M. Jones