September 2015 - Libguestfs - libguestfs List Archives

Plan for libguestfs 1.30
by Richard W.M. Jones 14 Dec '15

14 Dec '15

It has been a ludicrously long time (over 8 months) since the last stable release of libguestfs. So I'd like to plan a new 1.30 release soon. Please follow-up if there are features / blockers / bugs that need to be addressed for 1.30. As usual, bugs which have "1.30" (without quotes) in the Devel Whiteboard field in Bugzilla are nominated as blockers for the release. You can see a list of those here [currently empty]: https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&… For a list of all bugs, go to: https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtua… ---------------------------------------------------------------------- It's also worth talking about what I'd like to do *after* 1.30 is out. This includes: - threading support https://www.redhat.com/archives/libguestfs/2015-June/thread.html#00118 - external tests https://www.redhat.com/archives/libguestfs/2014-October/thread.html#00042 While thread safety is oft talked about, it's external tests which I consider to be the more important patch series, since it will allow us to build and test libguestfs more easily and more comprehensively for Fedora and RHEL. Unfortunately it's the most difficult and invasive change of the two. You may have your own ideas, if so follow up here. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

3 6

SELinux relabel API
by Richard W.M. Jones 10 Dec '15

10 Dec '15

[ I realized that we were discussing adding this feature, in various private email, IRC, and this long bugzilla thread: https://bugzilla.redhat.com/show_bug.cgi?id=1060423 That's not how we should do things. Let's discuss it on the mailing list. ] One thing that virt-customize/virt-sysprep/virt-builder have to do is relabel SELinux guests. What we do at the moment is run: if load_policy && fixfiles restore; then rm -f /.autorelabel else touch /.autorelabel echo '%s: SELinux relabelling failed, will relabel at boot instead.' fi while chrooted into the guest (using the 'guestfs_sh' API). This has a number of problems: - It has to load the policy using 'load_policy', but this doesn't work sometimes: * RHEL 5 load_policy takes a parameter. * Doesn't work if appliance kernel is significantly different from guest kernel version, because the binary policy format changes irregularly and is not backwards compatible. * Requires the appliance [host] kernel to be compiled with LSM/SELinux support. - Touching /.autorelabel is often broken, eg. it's broken in Fedora 20 because of systemd (RHBZ#1049656). - /etc/resolv.conf will not be relabelled if guestfs network is on, because of resolv.conf shenanigans in libguestfs.git/daemon/command.c - It requires running guest code, which we'd like to avoid. What would be nice would be to have an API to just do this relabelling. Libguestfs could change this API as required to handle different guests. Dan Walsh helpfully pointed out to us that we've been doing it wrong all along :-) A much better way to relabel is to run: setfiles /etc/selinux/targeted/contexts/files/file_contexts DIR where 'file_contexts' is a file which contains the default labels for files (a set of regexps), and 'DIR' is the directory at which relabelling starts. Note that 'setfiles' would be the libguestfs appliance binary, so no guest binary needs to be run. A simple API could just look like this: guestfs_selinux_relabel (g); which would always use the 'targeted' policy from the guest, and always start relabelling at the root. This would work fine for virt-builder. For Colin's requirements for Project Atomic, I suspect he will want to be able to set the file_contexts file and the root directory, but I'll leave him to describe what would be useful. A couple of notes: - I'd like to avoid baking in assumptions from the 'setfiles' command as far as possible. libguestfs APIs last for many years and some have caused us many years of regret (but that's our job) :-/ - Is it a good idea to tie this into inspection in some way -- for example, inspection could provide us with the path to the current or default SELinux policy. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

5 10

[PATCH] v2v: add --in-place mode
by Roman Kagan 20 Oct '15

20 Oct '15

In this mode, converting of the VM configuration, setting up the rollback path for error cases, transforming the VM storage and so on is taken care of by a third-party toolset, and virt-v2v is only supposed to tune up the guest OS directly inside the source VM, to enable it to boot and run under the input hypervisor. Signed-off-by: Roman Kagan <rkagan(a)virtuozzo.com> --- tests/guests/guests.xml.in | 16 +++++++ v2v/Makefile.am | 1 + v2v/cmdline.ml | 7 +++- v2v/test-v2v-in-place.sh | 81 +++++++++++++++++++++++++++++++++++ v2v/v2v.ml | 102 +++++++++++++++++++++++++++------------------ v2v/virt-v2v.pod | 17 ++++++++ 6 files changed, 183 insertions(+), 41 deletions(-) create mode 100755 v2v/test-v2v-in-place.sh diff --git a/tests/guests/guests.xml.in b/tests/guests/guests.xml.in index 8f7ac81..6f08b80 100644 --- a/tests/guests/guests.xml.in +++ b/tests/guests/guests.xml.in @@ -279,4 +279,20 @@ </devices> </domain> + <domain type='test'> + <name>windows-overlay</name> + <memory>1048576</memory> + <os> + <type>hvm</type> + <boot dev='hd'/> + </os> + <devices> + <disk type='file' device='disk'> + <driver name='qemu' type='qcow2'/> + <source file='@abs_builddir@/windows-overlay.qcow2'/> + <target dev='vda' bus='virtio'/> + </disk> + </devices> + </domain> + </node> diff --git a/v2v/Makefile.am b/v2v/Makefile.am index 06da002..dae063c 100644 --- a/v2v/Makefile.am +++ b/v2v/Makefile.am @@ -232,6 +232,7 @@ TESTS += \ test-v2v-cdrom.sh \ test-v2v-i-ova.sh \ test-v2v-i-disk.sh \ + test-v2v-in-place.sh \ test-v2v-machine-readable.sh \ test-v2v-networks-and-bridges.sh \ test-v2v-no-copy.sh \ diff --git a/v2v/cmdline.ml b/v2v/cmdline.ml index eaf57dc..2a3224a 100644 --- a/v2v/cmdline.ml +++ b/v2v/cmdline.ml @@ -37,6 +37,7 @@ let parse_cmdline () = let output_format = ref "" in let output_name = ref "" in let output_storage = ref "" in + let in_place = ref false in let password_file = ref "" in let print_source = ref false in let qemu_boot = ref false in @@ -160,6 +161,7 @@ let parse_cmdline () = "-of", Arg.Set_string output_format, "raw|qcow2 " ^ s_"Set output format"; "-on", Arg.Set_string output_name, "name " ^ s_"Rename guest when converting"; "-os", Arg.Set_string output_storage, "storage " ^ s_"Set output storage location"; + "--in-place", Arg.Set in_place, " " ^ s_"Only tune the guest in the input VM"; "--password-file", Arg.Set_string password_file, "file " ^ s_"Use password from file"; "--print-source", Arg.Set print_source, " " ^ s_"Print source and stop"; "--qemu-boot", Arg.Set qemu_boot, " " ^ s_"Boot in qemu (-o qemu only)"; @@ -226,6 +228,7 @@ read the man page virt-v2v(1). let output_mode = !output_mode in let output_name = match !output_name with "" -> None | s -> Some s in let output_storage = !output_storage in + let in_place = !in_place in let password_file = match !password_file with "" -> None | s -> Some s in let print_source = !print_source in let qemu_boot = !qemu_boot in @@ -305,6 +308,8 @@ read the man page virt-v2v(1). Input_ova.input_ova filename in (* Parse the output mode. *) + if output_mode <> `Not_set && in_place then + error (f_"-o and --in-place cannot be used at the same time"); let output = match output_mode with | `Glance -> @@ -386,5 +391,5 @@ read the man page virt-v2v(1). input, output, debug_gc, debug_overlays, do_copy, network_map, no_trim, - output_alloc, output_format, output_name, + output_alloc, output_format, output_name, in_place, print_source, root_choice diff --git a/v2v/test-v2v-in-place.sh b/v2v/test-v2v-in-place.sh new file mode 100755 index 0000000..c19707e --- /dev/null +++ b/v2v/test-v2v-in-place.sh @@ -0,0 +1,81 @@ +#!/bin/bash - +# libguestfs virt-v2v test script +# Copyright (C) 2014 Red Hat Inc. +# Copyright (C) 2015 Parallels IP Holdings GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Test --in-place. + +unset CDPATH +export LANG=C +set -e + +if [ -n "$SKIP_TEST_V2V_IN_PLACE_SH" ]; then + echo "$0: test skipped because environment variable is set" + exit 77 +fi + +if [ "$(guestfish get-backend)" = "uml" ]; then + echo "$0: test skipped because UML backend does not support network" + exit 77 +fi + +# You shouldn't be running the tests as root anyway, but in this case +# it's especially bad because we don't want to start creating guests +# or storage pools in the system namespace. +if [ "$(id -u)" -eq 0 ]; then + echo "$0: test skipped because you're running tests as root. Don't do that!" + exit 77 +fi + +guests_dir="$(cd $(dirname $0)/../tests/guests && pwd)" +libvirt_uri="test://$guests_dir/guests.xml" + +f="$guests_dir/windows.img" +if ! test -f $f || ! test -s $f; then + echo "$0: test skipped because phony Windows image was not created" + exit 77 +fi + +virt_tools_data_dir=${VIRT_TOOLS_DATA_DIR:-/usr/share/virt-tools} +if ! test -r $virt_tools_data_dir/rhsrvany.exe; then + echo "$0: test skipped because rhsrvany.exe is not installed" + exit 77 +fi + +fo="$guests_dir/windows-overlay.qcow2" +rm -f $fo +qemu-img create -f qcow2 -b $f -o compat=1.1,backing_fmt=raw $fo +md5="$(md5sum $f)" + +$VG virt-v2v --debug-gc \ + -i libvirt -ic "$libvirt_uri" windows-overlay \ + --in-place + +# Test some aspects of the target disk image. +guestfish --ro -a $fo -i <<EOF + is-dir "/Program Files/Red Hat/Firstboot" + is-file "/Program Files/Red Hat/Firstboot/firstboot.bat" + is-dir "/Program Files/Red Hat/Firstboot/scripts" + is-dir "/Windows/Drivers/VirtIO" +EOF + + +# Test the base image remained untouched +test "$md5" = "$(md5sum $f)" + +# Clean up. +rm -r $fo diff --git a/v2v/v2v.ml b/v2v/v2v.ml index 242f129..b744056 100644 --- a/v2v/v2v.ml +++ b/v2v/v2v.ml @@ -47,7 +47,8 @@ let rec main () = (* Handle the command line. *) let input, output, debug_gc, debug_overlays, do_copy, network_map, no_trim, - output_alloc, output_format, output_name, print_source, root_choice = + output_alloc, output_format, output_name, in_place, print_source, + root_choice = Cmdline.parse_cmdline () in (* Print the version, easier than asking users to tell us. *) @@ -117,52 +118,70 @@ let rec main () = ) nics in { source with s_nics = nics } in - (* Create a qcow2 v3 overlay to protect the source image(s). There - * is a specific reason to use the newer qcow2 variant: Because the - * L2 table can store zero clusters efficiently, and because - * discarded blocks are stored as zero clusters, this should allow us - * to fstrim/blkdiscard and avoid copying significant parts of the - * data over the wire. - *) - message (f_"Creating an overlay to protect the source from being modified"); let overlay_dir = (new Guestfs.guestfs ())#get_cachedir () in - let overlays = - List.map ( - fun ({ s_qemu_uri = qemu_uri; s_format = format } as source) -> - let overlay_file = - Filename.temp_file ~temp_dir:overlay_dir "v2vovl" ".qcow2" in - unlink_on_exit overlay_file; - - let options = - "compat=1.1" ^ - (match format with None -> "" - | Some fmt -> ",backing_fmt=" ^ fmt) in - let cmd = - sprintf "qemu-img create -q -f qcow2 -b %s -o %s %s" - (quote qemu_uri) (quote options) overlay_file in - if verbose () then printf "%s\n%!" cmd; - if Sys.command cmd <> 0 then - error (f_"qemu-img command failed, see earlier errors"); - - (* Sanity check created overlay (see below). *) - if not ((new G.guestfs ())#disk_has_backing_file overlay_file) then - error (f_"internal error: qemu-img did not create overlay with backing file"); - - overlay_file, source - ) source.s_disks in + let overlays = ( + if not in_place then ( + (* Create a qcow2 v3 overlay to protect the source image(s). There + * is a specific reason to use the newer qcow2 variant: Because the + * L2 table can store zero clusters efficiently, and because + * discarded blocks are stored as zero clusters, this should allow us + * to fstrim/blkdiscard and avoid copying significant parts of the + * data over the wire. + *) + message (f_"Creating an overlay to protect the source from being modified"); + List.map ( + fun ({ s_qemu_uri = qemu_uri; s_format = format } as source) -> + let overlay_file = + Filename.temp_file ~temp_dir:overlay_dir "v2vovl" ".qcow2" in + unlink_on_exit overlay_file; + + let options = + "compat=1.1" ^ + (match format with None -> "" + | Some fmt -> ",backing_fmt=" ^ fmt) in + let cmd = + sprintf "qemu-img create -q -f qcow2 -b %s -o %s %s" + (quote qemu_uri) (quote options) overlay_file in + if verbose () then printf "%s\n%!" cmd; + if Sys.command cmd <> 0 then + error (f_"qemu-img command failed, see earlier errors"); + + (* Sanity check created overlay (see below). *) + if not ((new G.guestfs ())#disk_has_backing_file overlay_file) then + error (f_"internal error: qemu-img did not create overlay with backing file"); + + overlay_file, source + ) source.s_disks + ) else [] + ) in (* Open the guestfs handle. *) - message (f_"Opening the overlay"); + if in_place then + message (f_"Opening the guest disks") + else + message (f_"Opening the overlay"); let g = new G.guestfs () in if trace () then g#set_trace true; if verbose () then g#set_verbose true; g#set_network true; - List.iter ( - fun (overlay_file, _) -> - g#add_drive_opts overlay_file - ~format:"qcow2" ~cachemode:"unsafe" ~discard:"besteffort" - ~copyonread:true - ) overlays; + if in_place then ( + List.iter ( + fun ({s_qemu_uri = qemu_uri; s_format = format}) -> + match format with + | None -> + g#add_drive_opts qemu_uri ~cachemode:"unsafe" ~discard:"besteffort" + | Some fmt -> + g#add_drive_opts qemu_uri ~format:fmt ~cachemode:"unsafe" + ~discard:"besteffort" + ) source.s_disks + ) else ( + List.iter ( + fun (overlay_file, _) -> + g#add_drive_opts overlay_file + ~format:"qcow2" ~cachemode:"unsafe" ~discard:"besteffort" + ~copyonread:true + ) overlays + ); g#launch (); @@ -294,6 +313,9 @@ let rec main () = g#shutdown (); g#close (); + if in_place then + exit 0; + (* Does the guest require UEFI on the target? *) message (f_"Checking if the guest needs BIOS or UEFI to boot"); let target_firmware = diff --git a/v2v/virt-v2v.pod b/v2v/virt-v2v.pod index ef2dee1..eda1cf7 100644 --- a/v2v/virt-v2v.pod +++ b/v2v/virt-v2v.pod @@ -9,6 +9,8 @@ virt-v2v - Convert a guest to use KVM virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \ -o rhev -os rhev.nfs:/export_domain --network rhevm + virt-v2v -ic qemu:///system qemu_guest --in-place + virt-v2v -i libvirtxml guest-domain.xml -o local -os /var/tmp virt-v2v -i disk disk.img -o local -os /var/tmp @@ -75,6 +77,9 @@ booting the guest directly in qemu (mainly for testing). I<-o rhev> is used to write to a RHEV-M / oVirt target. I<-o vdsm> is only used when virt-v2v runs under VDSM control. +I<--in-place> instructs virt-v2v to customize the guest OS in the input +virtual machine, instead of creating a new VM in the target hypervisor. + =head1 EXAMPLES =head2 Convert from VMware vCenter server to local libvirt @@ -518,6 +523,18 @@ C<root>. You will get an error if virt-v2v is unable to mount/write to the Export Storage Domain. +=item B<--in-place> + +Do not create an output virtual machine in the target hypervisor. +Instead, adjust the guest OS in the source VM to run in the input +hypervisor. + +This mode is meant for integration with other toolsets, which take the +responsibility of converting the VM configuration, providing for +rollback in case of errors, transforming the storage, etc. + +Conflicts with all I<-o *> options. + =item B<--password-file> file Instead of asking for password(s) interactively, pass the password -- 2.4.3

3 74

[PATCH 0/4] v2v: simplify driver copying from virtio-win iso
by Roman Kagan 15 Oct '15

15 Oct '15

Libguestfs supports passing an ISO image as a source of virtio windows drivers to v2v. That support, however, looks too heavy-weight: in order to access those drivers, a separate guestfs handle is created (and thus a new emulator process is started), which runs until v2v completes. This series attempts to make it simpler and lighter-weight, by making the relevant code more local, and by hot-adding the image into the main guestfs handle. Roman Kagan (4): v2v: drop useless forced gc v2v: consolidate virtio-win file copying v2v: copy virtio drivers without guestfs handle leak v2v: reuse main guestfs for virtio win drivers iso v2v/convert_windows.ml | 184 ++++++++++++++++++++-------------------- v2v/utils.ml | 224 ++++++++++++++++--------------------------------- v2v/v2v.ml | 8 -- 3 files changed, 163 insertions(+), 253 deletions(-) -- 2.4.3

2 39

[PATCH] Add opensuse.gpg to the dist tarball
by Cédric Bosdonnat 30 Sep '15

30 Sep '15

Adding missing opensuse.gpg to EXTRA_DIST. --- builder/Makefile.am | 1 + 1 file changed, 1 insertion(+) diff --git a/builder/Makefile.am b/builder/Makefile.am index 3111ef2..0d5ce3b 100644 --- a/builder/Makefile.am +++ b/builder/Makefile.am @@ -22,6 +22,7 @@ AM_YFLAGS = -d EXTRA_DIST = \ $(SOURCES_MLI) $(SOURCES_ML) $(SOURCES_C) \ libguestfs.gpg \ + opensuse.gpg \ test-index \ test-simplestreams/streams/v1/index.json \ test-simplestreams/streams/v1/net.cirros-cloud_released_download.json \ -- 2.1.4

1 0

[PATCH 1/2] ocaml: Use ocamlfind to run ocamldoc.
by Richard W.M. Jones 30 Sep '15

30 Sep '15

Using 'ocamlfind ocamldoc' is much faster than running 'ocamldoc' directly, because ocamlfind will run the native code program 'ocamldoc.opt' if it is available. This change approximately halves the time taken to compile the ocaml bindings. --- ocaml/Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ocaml/Makefile.am b/ocaml/Makefile.am index a535b43..b9828e1 100644 --- a/ocaml/Makefile.am +++ b/ocaml/Makefile.am @@ -98,7 +98,7 @@ if HAVE_OCAMLDOC noinst_DATA += html/index.html html/index.html: $(srcdir)/guestfs.mli $(srcdir)/guestfs.ml - -$(OCAMLDOC) -d html -html $^ + -$(OCAMLFIND) ocamldoc -d html -html $^ endif -- 2.5.0

2 4

[PATCH 1/4] lib: actions: Remove some unused header files.
by Richard W.M. Jones 30 Sep '15

30 Sep '15

--- generator/c.ml | 2 -- 1 file changed, 2 deletions(-) diff --git a/generator/c.ml b/generator/c.ml index 055b683..963cf21 100644 --- a/generator/c.ml +++ b/generator/c.ml @@ -1213,9 +1213,7 @@ and generate_client_actions hash () = #include <stdlib.h> #include <stdint.h> #include <inttypes.h> -#include <sys/types.h> #include <sys/stat.h> -#include <string.h> #include \"guestfs.h\" #include \"guestfs-internal.h\" -- 2.5.0

2 9

[PATCH] daemon: Compile stubs.c first.
by Richard W.M. Jones 30 Sep '15

30 Sep '15

This shaves about 20% off the compile time for the daemon subdirectory. See also: https://rwmj.wordpress.com/2015/09/30/make-and-queuing-theory/#content --- daemon/Makefile.am | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/daemon/Makefile.am b/daemon/Makefile.am index 1d29a0f..4ea3c88 100644 --- a/daemon/Makefile.am +++ b/daemon/Makefile.am @@ -76,7 +76,11 @@ else noinst_PROGRAMS = guestfsd endif +# Compile the largest file (stubs.c) first. The other files are +# listed alphabetically. See also: +# https://rwmj.wordpress.com/2015/09/30/make-and-queuing-theory/#content guestfsd_SOURCES = \ + stubs.c \ 9p.c \ acl.c \ actions.h \ @@ -162,7 +166,6 @@ guestfsd_SOURCES = \ stat.c \ statvfs.c \ strings.c \ - stubs.c \ swap.c \ sync.c \ syslinux.c \ -- 2.5.0

2 1

[PATCH 1/2] ocaml: Only build the tests when running 'make check'.
by Richard W.M. Jones 30 Sep '15

30 Sep '15

Make the tests 'check_DATA' so they only get built when running the tests. This saves a couple of seconds on the build time. --- ocaml/Makefile.am | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ocaml/Makefile.am b/ocaml/Makefile.am index e781363..a535b43 100644 --- a/ocaml/Makefile.am +++ b/ocaml/Makefile.am @@ -142,9 +142,9 @@ endif TESTS = run-bindtests $(test_progs_all) -noinst_DATA += bindtests.bc $(test_progs_all) +check_DATA = bindtests.bc $(test_progs_all) if HAVE_OCAMLOPT -noinst_DATA += bindtests.opt +check_DATA += bindtests.opt endif %.bc: %.cmo mlguestfs.cma @@ -204,6 +204,6 @@ install-data-hook: rm $(DESTDIR)$(OCAMLLIB)/guestfs/bindtests.* rm $(DESTDIR)$(OCAMLLIB)/guestfs/libguestfsocaml.a -CLEANFILES += $(noinst_DATA) +CLEANFILES += $(noinst_DATA) $(check_DATA) endif -- 2.5.0

2 2

[PATCH v2] copy-in: error out early if the localpath does not exist
by Pino Toscano 30 Sep '15

30 Sep '15

--- src/copy-in-out.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/copy-in-out.c b/src/copy-in-out.c index d5e7fb0..89344d8 100644 --- a/src/copy-in-out.c +++ b/src/copy-in-out.c @@ -46,6 +46,12 @@ guestfs_impl_copy_in (guestfs_h *g, size_t buf_len = strlen (localpath) + 1; char buf[buf_len]; const char *dirname, *basename; + struct stat statbuf; + + if (stat (localpath, &statbuf) == -1) { + error (g, _("source '%s' does not exist (or cannot be read)"), localpath); + return -1; + } int remote_is_dir = guestfs_is_dir (g, remotedir); if (remote_is_dir == -1) -- 2.1.0

2 1