August 2010 - Libguestfs - libguestfs List Archives

[PATCH v3 0/5] Inspection code in C
by Richard W.M. Jones 17 Aug '10

17 Aug '10

The first three patches were posted previously: https://www.redhat.com/archives/libguestfs/2010-July/msg00082.html The last two patches in this series change guestfish -i to use this new code. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

2 8

[PATCH] Python: Use new PyCapsule API where supported.
by Richard W.M. Jones 17 Aug '10

17 Aug '10

I have run the automated tests on two machines, with Python 2.6 and 2.7 respectively, and at least it didn't fail on either. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v

1 0

Plans for changing libguestfs appliance building
by Richard W.M. Jones 17 Aug '10

17 Aug '10

Well you can all read the thread here for the technical details: http://lists.gnu.org/archive/html/qemu-devel/2010-08/threads.html#00133 but the bottom line for anyone who wasn't in that discussion or on IRC last week is that we have to change the way that the appliance is built in libguestfs so that we don't depend (as much) on the qemu -initrd option. The bad thing is this is loads of make-work at a particularly inconvenient time. The good thing is that it should make libguestfs 'boot' in under 5 seconds, even if your hardware is old and slow. It will also reduce the memory and I/O requirements for using libguestfs. Because this issue is adversely affecting people using kernel 2.6.35 and/or the latest qemu right now (and will affect many people when Fedora 14 is released which has both), this is my top priority to fix in the coming week. This email documents how I will change appliance building. The patches will follow some time later in the week when I've written and tested them. The new boot method will be this sequence: (a) (Same as now) A suitable kernel to use is located on the host, or from $libdir/guestfs in the non-supermin case. (b) A tiny initrd is built (on the fly for supermin). This will contain just these files: any CD-ROM, IDE, virtio kmods required to read the virtual CD device ext2.ko if required /sbin/modprobe a tiny init script, written in C (c) An ext2 image is created for the root filesystem. I will say a lot more about how this is created below. (d) qemu is invoked with something along these lines: qemu -kernel kernel -initrd initrd -drive file=isofile,media=cdrom where: kernel = kernel found in step (a) initrd = tiny initrd created in step (b) isofile = ext2 filesystem image created in step (c) (e) The boot process proceeds by starting the kernel, reading the /init script from the tiny initrd which mounts the filesystem from the CD-ROM device and pivot_root(2)s into it, running another /init script from that filesystem. At this point, boot continues as it does in the current libguestfs. Since this involves attaching an extra device to the appliance, we also need to change the daemon to ignore this extra device, adding somewhat to the complexity of several operations. We believe this should improve the speed of boot greatly. Obviously there is the saving because we are no longer using the (now broken) qemu -initrd support with a large initrd, but that's purely because of a qemu regression. We also save because we don't need to unpack the initrd inside the appliance, and because the device is loaded on demand. Also there are some unrelated changes that I intend to make which will improve boot speeds. We have every reason to believe that 5 seconds will be achievable, even on relatively old hardware. Step (c) above involves creating an ext2 filesystem for root. I chose ext2 because it is considerably less complex than ISO9660, and is the native Linux filesystem so it supports all Linux features (long filenames, extended attributes and so on). Creating an ext2 filesystem is more complex than the initrd that we currently create using some hand-coded cpio-like code. Of course we can't use libguestfs to help us! We plan to write some C code to assemble the ext2 filesystem from scratch (or starting from a mke2fs blank filesystem). To further improve boot times, we intend to cache this so it will only be created the first time it is used, and only when it needs to be updated. Caching this means that most times that you use libguestfs or other tools, no appliance creation will be required at all, and only the bits of the appliance that you use will be loaded. If you have any further questions about this, please follow up on this mailing list. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top

5 11

[PATCH 1/2] Allow absolute paths in virt-v2v.conf
by Matthew Booth 16 Aug '10

16 Aug '10

This patch allows paths in virt-v2v.conf to be either relative or absolute. If relative, they are relative to software-root. This allows virt-v2v.conf to use files provided by packages independent of virt-v2v. --- lib/Sys/VirtV2V/Config.pm | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/lib/Sys/VirtV2V/Config.pm b/lib/Sys/VirtV2V/Config.pm index f703152..121e774 100644 --- a/lib/Sys/VirtV2V/Config.pm +++ b/lib/Sys/VirtV2V/Config.pm @@ -118,14 +118,16 @@ sub get_transfer_iso foreach my $path ($dom->findnodes('/virt-v2v/app/path/text()')) { $path = $path->getData(); - # Get the absolute path if iso-root was defined my $abs; - if (defined($root)) { - $abs = File::Spec->catfile($root, $path); - } else { + if (File::Spec->file_name_is_absolute($path) || !defined($root)) { $abs = $path; } + # Make relative paths relative to iso-root if it was defined + else { + $abs = File::Spec->catfile($root, $path); + } + if (-r $abs) { $path_args{"$path=$abs"} = 1; $paths{$abs} = 1; -- 1.7.2.1

2 4

[PATCH] Install VirtIO storage and network drivers in Windows
by Matthew Booth 16 Aug '10

16 Aug '10

Currently when converting a Windows guest we do a minimum installation of the viostor driver, configure the RHEV guest agent and leave RHEV to properly install viostor and all remaining drivers. This works well if RHEV is properly configured and the installation is not interrupted on first boot. However, if the target of the conversion is not RHEV, RHEV is not properly configured, or the first boot installation process is interrupted, for example by the user logging in and interacting with it, this will fail. In this case, in the absence of a correct driver Windows can mis-detect the VirtIO 'SCSI Controller' and configure the wrong driver for it. This will lead to the guest subsequently failing to boot. This patch complements the RHEV-managed process by additionally copying installable versions of the VirtIO storage and network drivers to the guest during conversion, and adding the location of the drivers to the default search path for drivers. This means that Windows will install correct drivers for network and storage if the RHEV process fails, or if the conversion target is not RHEV. --- lib/Sys/VirtV2V/Converter/Windows.pm | 77 ++++++++++++++++++++++++++++++++++ 1 files changed, 77 insertions(+), 0 deletions(-) diff --git a/lib/Sys/VirtV2V/Converter/Windows.pm b/lib/Sys/VirtV2V/Converter/Windows.pm index 1d4c526..f5bf399 100644 --- a/lib/Sys/VirtV2V/Converter/Windows.pm +++ b/lib/Sys/VirtV2V/Converter/Windows.pm @@ -23,6 +23,7 @@ use warnings; use Carp qw(carp); use File::Temp qw(tempdir); use Data::Dumper; +use Encode qw(encode decode); use IO::String; use XML::DOM; use XML::DOM::XPath; @@ -187,6 +188,7 @@ sub _preconvert _upload_files ($g, $tmpdir, $desc, $devices, $config); _add_viostor_to_registry ($g, $tmpdir, $desc, $devices, $config); _add_service_to_registry ($g, $tmpdir, $desc, $devices, $config); + _prepare_virtio_drivers ($g, $tmpdir, $desc, $devices, $config); } # See http://rwmj.wordpress.com/2010/04/30/tip-install-a-device-driver-in-a-windo… @@ -345,6 +347,81 @@ sub _add_service_to_registry $g->upload ($tmpdir . "/system", $system_filename); } +# We copy the VirtIO drivers to a directory on the guest and add this directory +# to HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\DevicePath so that it will +# be searched automatically when automatically installing drivers. +sub _prepare_virtio_drivers +{ + my $g = shift; + my $tmpdir = shift; + my $desc = shift; + my $devices = shift; + my $config = shift; + + # Copy the target VirtIO drivers to the guest + my $driverdir = File::Spec->catdir($g->case_sensitive_path("/windows"), "Drivers/VirtIO"); + + $g->mkdir_p($driverdir); + + my ($virtio) = $config->match_app ($desc, 'virtio', $desc->{arch}); + $virtio = $config->get_transfer_path($g, $virtio); + + foreach my $src ($g->ls($virtio)) { + my $name = $src; + $src = File::Spec->catfile($virtio); + my $dst = File::Spec->catfile($driverdir, $name); + $g->cp_a($src, $dst); + } + + # Locate and download the SOFTWARE hive + my $sw_local = File::Spec->catfile($tmpdir, 'software'); + my $sw_guest = $g->case_sensitive_path('/windows/system32/config/software'); + + $g->download($sw_guest, $sw_local); + + # Open the registry hive. + my $h = Win::Hivex->open($sw_local, write => 1) + or die "open hive $sw_local: $!"; + + # Find the node \Microsoft\Windows\CurrentVersion + my $node = $h->root(); + foreach ('Microsoft', 'Windows', 'CurrentVersion') { + $node = $h->node_get_child($node, $_); + } + + # Update DevicePath, but leave everything else as is + my @new; + my $append = ';%SystemRoot%\Drivers\VirtIO'; + foreach my $v ($h->node_values($node)) { + my $key = $h->value_key($v); + my ($type, $data) = $h->value_value($v); + + # Decode the string from utf16le to perl native + my $value = decode('UTF-16LE', $data); + + # Append the driver location if it's not there already + if ($key eq 'DevicePath' && index($value, $append) == -1) { + # Remove the explicit trailing NULL + chop($value); + + # Append the new path and a new explicit trailing NULL + $value .= $append."\0"; + + # Re-encode the string back to utf16le + $data = encode('UTF-16LE', $value); + } + + push (@new, { key => $key, t => $type, value => $data }); + } + $h->node_set_values($node, \@new); + + $h->commit(undef); + undef $h; + + # Upload the new registry. + $g->upload($sw_local, $sw_guest); +} + sub _upload_files { my $g = shift; -- 1.7.2.1

2 1

[PATCH] Add debug output to hivex_close
by Matthew Booth 16 Aug '10

16 Aug '10

--- lib/hivex.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/lib/hivex.c b/lib/hivex.c index 13d7556..8a774de 100644 --- a/lib/hivex.c +++ b/lib/hivex.c @@ -543,6 +543,9 @@ hivex_close (hive_h *h) free (h->filename); free (h); + if (h->msglvl >= 1) + fprintf (stderr, "hivex_close\n"); + return r; } -- 1.7.2.1

2 3

Question about using guestfish --ro as a backup solution
by Richard W.M. Jones 16 Aug '10

16 Aug '10

> <monolive> what is the risk if I backup a running VM ? using > guestfish --ro ? I understand that my open files might be funny but > the issue should be solve by a fsck ? it won't work for an open DB It's a bit different from this. What happens is that the libguestfs appliance / kernel attaches to the disk, which is in a potentially unclean *and* potentially changing state. When libguestfs does the mount, the journal recovery is performed (against a throw-away snapshot of the original disk -- if you use the --ro option, the original disk is not written to at all). However the disk is still changing and the libguestfs kernel could interpret this in all sorts of ways: eg. panicking or silently reading corrupt data. The news is: DON'T use guestfish --ro as your backup solution, UNLESS: (1) you don't care about the integrity of your backups, or (2) you can prove that nothing is writing to the disk (eg. the VM is switched off), or (3) you take a snapshot of the disk first (eg. LVM snapshot), which is really just a special case of (2), or (4) the filesystem that you are backing up is frozen[a], which is also a special case of (2). Run a backup daemon inside the guest instead. There are plenty of network backup programs around. Choose your favorite one and install it in your VM. Rich. [a] http://lwn.net/Articles/287435/ -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/

2 1

Question about blkdev-setbsz
by Richard W.M. Jones 13 Aug '10

13 Aug '10

> <jzheng> Do you have any idea why blockdev-setbsz succeeds, but does > not affect output of blockdev-getbsz? I can reproduce the same issue too: ><fs> blockdev-getbsz /dev/vda send_to_daemon: 0xf62050 g->state = 3, n = 40 recv_from_daemon: 0xf62050 g->state = 3, size_rtn = 0x7ffffbb4f00c, buf_rtn = 0x7ffffbb4f000 blockdev --getbsz /dev/vda proc 60 (blockdev_getbsz) took 0.01 seconds 4096 ><fs> blockdev-setbsz /dev/vda 2048 send_to_daemon: 0xf62050 g->state = 3, n = 44 recv_from_daemon: 0xf62050 g->state = 3, size_rtn = 0x7ffffbb4f01c, buf_rtn = 0x7ffffbb4f010 blockdev --setbsz 2048 /dev/vda proc 61 (blockdev_setbsz) took 0.01 seconds ><fs> blockdev-getbsz /dev/vda send_to_daemon: 0xf62050 g->state = 3, n = 40 recv_from_daemon: 0xf62050 g->state = 3, size_rtn = 0x7ffffbb4f00c, buf_rtn = 0x7ffffbb4f000 blockdev --getbsz /dev/vda proc 60 (blockdev_getbsz) took 0.01 seconds 4096 I checked the source of blockdev, and it is using the BLKBSZSET ioctl. As far as I can tell if the ioctl returns an error then blockdev should return an error too (and hence the error should go all the way out to guestfish). So it appears that the Linux kernel is not returning an error. http://lxr.linux.no/#linux+v2.6.35/fs/block_dev.c#L73 The other possibility is that the blockdev --getbsz call is not returning the block size we just set, but that resolves to this call inside the kernel: http://lxr.linux.no/#linux+v2.6.35/include/linux/blkdev.h#L1196 As far as I can see this really ought to work. Perhaps we need to perform some other operation between the two calls, like a udev settle or some sort of rereading block devices ioctl. If this worries you, please file a BZ. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

1 0

Re: [Libguestfs] guestfish/libguestfs takes legacy qemu instead of kvm?
by Richard W.M. Jones 10 Aug '10

10 Aug '10

On Sun, Aug 08, 2010 at 12:05:58PM +0800, Kirby Zhou wrote: > guestfish takes legacy qemu instead of kvm? > ]# rpm -q libguestfs --requires | fgrep qemu > qemu-system-x86 >= 0.10.5 To make it clear, this is EPEL-5. There was some still unresolved problem with adding a dependency on KVM: https://fedorahosted.org/rel-eng/ticket/2982 As a result, we have to depend on qemu (software emulation) instead. However you can easily change this at runtime by setting the LIBGUESTFS_QEMU environment variable: http://libguestfs.org/guestfs.3.html#libguestfs_qemu http://libguestfs.org/guestfs.3.html#qemu_wrappers In RHEL 6 it's all going to be different again. The same qemu-system-x86 RPM is actually KVM. We combine qemu and KVM sources during the build to make a 'qemu-kvm' binary which can do both software emulation (TCG) and hardware virt (KVM). > [@djt_10_48 ~]# guestfish -i kvm-rhel5guest > Could not open '/dev/kqemu' - QEMU acceleration layer not activated: No such > file or directory This is just a warning -- ignore it. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/

3 5

[PATCH 0/3] Inspection code in C
by Richard W.M. Jones 02 Aug '10

02 Aug '10

These three patches (two were previously posted) can do simple operating system inspection in C. Example of use: ><fs> add-ro rhel55.img ><fs> run ><fs> inspect-os /dev/VolGroup00/LogVol00 ><fs> inspect-get-type /dev/VolGroup00/LogVol00 linux ><fs> inspect-get-distro /dev/VolGroup00/LogVol00 rhel ><fs> inspect-get-arch /dev/VolGroup00/LogVol00 x86_64 ><fs> inspect-get-major-version /dev/VolGroup00/LogVol00 5 ><fs> inspect-get-minor-version /dev/VolGroup00/LogVol00 5 ><fs> inspect-get-product-name /dev/VolGroup00/LogVol00 Red Hat Enterprise Linux Server release 5.5 (Tikanga) ><fs> inspect-get-filesystems /dev/VolGroup00/LogVol00 /dev/VolGroup00/LogVol00 /dev/vda1 /dev/VolGroup00/LogVol01 ><fs> inspect-get-mountpoints /dev/VolGroup00/LogVol00 /dev/VolGroup00/LogVol00: / /dev/vda1: /boot I've tested this with all of my available guests. The only ones it *doesn't* work on were a Fedora guest with an encrypted disk, and some FreeBSD/PCBSD guests. For the encrypted guest, you have to open the device first. We could probably get the *BSD guests working with some effort. Anyway I'm quite happy with this and I intend to push it if no one objects. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

1 4