[PATCH] New APIs: {compress, gzip, xz}-out, {compress, gzip, xz}-device-out.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
This updates the existing patch by removing 'compress'. It turns out this program isn't installed in the appliance, and in any case the compress format is long obsolete. Below are some timings. Note that even for plain gzip compression there is quite a large time penalty. Virtio-serial / the libguestfs protocol is quite efficient at copying large files around (75 megabytes/sec, only a fraction slower than 'dd' on the host). There is no multithread support in gzip or xz at the moment, so adding SMP support to the appliance probably won't make much difference, although it's worth adding it anyway. I might play with snappy and a simple RLE-based compressor next. Rich. # Newly created, empty, 100MB ext2 filesystem: $ ./run ./fish/guestfish -N fs time download /dev/vda1 >(wc -c) elapsed time: 0.65 seconds 104792576 $ ./run ./fish/guestfish -N fs time gzip-device-out /dev/vda1 >(wc -c) elapsed time: 1.08 seconds 106799 $ ./run ./fish/guestfish -N fs time xz-device-out /dev/vda1 >(wc -c) elapsed time: 9.83 seconds 18168 # On a real filesystem that is too large to fit into cache: # Filesystem Size Used Available Use% # F16x64:/dev/vg_f16x64/lv_root 5.5G 3.2G 2.2G 59% $ ./run ./fish/guestfish --ro -a /dev/vg_pin/F16x64 run : time download /dev/vg_f16x64/lv_root >(wc -c) elapsed time: 78.74 seconds 5972688896 $ ./run ./fish/guestfish --ro -a /dev/vg_pin/F16x64 run : time gzip-device-out /dev/vg_f16x64/lv_root >(wc -c) elapsed time: 274.11 seconds 1927316373 [xz case took more than 10 minutes, I didn't wait for it to finish] -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
This third version changes the API again to allow you to select both the compression program (from {compress,gzip,bzip2,xz,lzop}) and optionally the compression level. Some more timings are below. 'lzop' is generally much faster than other methods, although there is still > 50% penalty for using lzop on a fairly full filesystem. Interestingly, 'lzop -1' which is supposed to be fast, is slower than the default level. Rich. 100 MB empty filesystem. Straight download was 0.65 seconds. $ ./run ./fish/guestfish -N fs \ time compress-device-out gzip /dev/vda1 >(wc -c) level:1 elapsed time: 0.92 seconds 462390 $ ./run ./fish/guestfish -N fs \ time compress-device-out gzip /dev/vda1 >(wc -c) level:9 elapsed time: 1.14 seconds 107903 $ ./run ./fish/guestfish -N fs \ time compress-device-out lzop /dev/vda1 >(wc -c) elapsed time: 0.73 seconds 428023 $ ./run ./fish/guestfish -N fs \ time compress-device-out lzop /dev/vda1 >(wc -c) level:1 elapsed time: 0.84 seconds 428125 $ ./run ./fish/guestfish -N fs \ time compress-device-out bzip2 /dev/vda1 >(wc -c) elapsed time: 1.80 seconds 2801 $ ./run ./fish/guestfish -N fs \ time compress-device-out xz /dev/vda1 >(wc -c) elapsed time: 9.17 seconds 18164 Real filesystem image ~5.5GB in size, 59% in use. Straight download was 78.74 seconds. $ ./run ./fish/guestfish --ro -a /dev/vg_pin/F16x64 \ run : time compress-device-out lzop /dev/vg_f16x64/lv_root >(wc -c) elapsed time: 123.17 seconds 2488500746 -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Wed, Sep 28, 2011 at 01:51:18PM +0100, Richard W.M. Jones wrote: [...] I have pushed this patch, and a further patch to enable SMP in the appliance (optional, default is still UP). Interestingly, enabling more virtual CPUs in the appliance causes a substantial slowdown in lzop performance. The following test:
Real filesystem image ~5.5GB in size, 59% in use. Straight download was 78.74 seconds.
$ ./run ./fish/guestfish --ro -a /dev/vg_pin/F16x64 \ run : time compress-device-out lzop /dev/vg_f16x64/lv_root >(wc -c) elapsed time: 123.17 seconds 2488500746
takes 148 seconds with "qemu-kvm -smp 4". Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
participants (1)
-
Richard W.M. Jones