July 2015 - Libguestfs - libguestfs List Archives

Plan for libguestfs 1.30
by Richard W.M. Jones 14 Dec '15

14 Dec '15

It has been a ludicrously long time (over 8 months) since the last stable release of libguestfs. So I'd like to plan a new 1.30 release soon. Please follow-up if there are features / blockers / bugs that need to be addressed for 1.30. As usual, bugs which have "1.30" (without quotes) in the Devel Whiteboard field in Bugzilla are nominated as blockers for the release. You can see a list of those here [currently empty]: https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&… For a list of all bugs, go to: https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtua… ---------------------------------------------------------------------- It's also worth talking about what I'd like to do *after* 1.30 is out. This includes: - threading support https://www.redhat.com/archives/libguestfs/2015-June/thread.html#00118 - external tests https://www.redhat.com/archives/libguestfs/2014-October/thread.html#00042 While thread safety is oft talked about, it's external tests which I consider to be the more important patch series, since it will allow us to build and test libguestfs more easily and more comprehensively for Fedora and RHEL. Unfortunately it's the most difficult and invasive change of the two. You may have your own ideas, if so follow up here. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

3 6

SELinux relabel API
by Richard W.M. Jones 10 Dec '15

10 Dec '15

[ I realized that we were discussing adding this feature, in various private email, IRC, and this long bugzilla thread: https://bugzilla.redhat.com/show_bug.cgi?id=1060423 That's not how we should do things. Let's discuss it on the mailing list. ] One thing that virt-customize/virt-sysprep/virt-builder have to do is relabel SELinux guests. What we do at the moment is run: if load_policy && fixfiles restore; then rm -f /.autorelabel else touch /.autorelabel echo '%s: SELinux relabelling failed, will relabel at boot instead.' fi while chrooted into the guest (using the 'guestfs_sh' API). This has a number of problems: - It has to load the policy using 'load_policy', but this doesn't work sometimes: * RHEL 5 load_policy takes a parameter. * Doesn't work if appliance kernel is significantly different from guest kernel version, because the binary policy format changes irregularly and is not backwards compatible. * Requires the appliance [host] kernel to be compiled with LSM/SELinux support. - Touching /.autorelabel is often broken, eg. it's broken in Fedora 20 because of systemd (RHBZ#1049656). - /etc/resolv.conf will not be relabelled if guestfs network is on, because of resolv.conf shenanigans in libguestfs.git/daemon/command.c - It requires running guest code, which we'd like to avoid. What would be nice would be to have an API to just do this relabelling. Libguestfs could change this API as required to handle different guests. Dan Walsh helpfully pointed out to us that we've been doing it wrong all along :-) A much better way to relabel is to run: setfiles /etc/selinux/targeted/contexts/files/file_contexts DIR where 'file_contexts' is a file which contains the default labels for files (a set of regexps), and 'DIR' is the directory at which relabelling starts. Note that 'setfiles' would be the libguestfs appliance binary, so no guest binary needs to be run. A simple API could just look like this: guestfs_selinux_relabel (g); which would always use the 'targeted' policy from the guest, and always start relabelling at the root. This would work fine for virt-builder. For Colin's requirements for Project Atomic, I suspect he will want to be able to set the file_contexts file and the root directory, but I'll leave him to describe what would be useful. A couple of notes: - I'd like to avoid baking in assumptions from the 'setfiles' command as far as possible. libguestfs APIs last for many years and some have caused us many years of regret (but that's our job) :-/ - Is it a good idea to tie this into inspection in some way -- for example, inspection could provide us with the path to the current or default SELinux policy. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

5 10

[PATCH] v2v: add --in-place mode
by Roman Kagan 20 Oct '15

20 Oct '15

In this mode, converting of the VM configuration, setting up the rollback path for error cases, transforming the VM storage and so on is taken care of by a third-party toolset, and virt-v2v is only supposed to tune up the guest OS directly inside the source VM, to enable it to boot and run under the input hypervisor. Signed-off-by: Roman Kagan <rkagan(a)virtuozzo.com> --- tests/guests/guests.xml.in | 16 +++++++ v2v/Makefile.am | 1 + v2v/cmdline.ml | 7 +++- v2v/test-v2v-in-place.sh | 81 +++++++++++++++++++++++++++++++++++ v2v/v2v.ml | 102 +++++++++++++++++++++++++++------------------ v2v/virt-v2v.pod | 17 ++++++++ 6 files changed, 183 insertions(+), 41 deletions(-) create mode 100755 v2v/test-v2v-in-place.sh diff --git a/tests/guests/guests.xml.in b/tests/guests/guests.xml.in index 8f7ac81..6f08b80 100644 --- a/tests/guests/guests.xml.in +++ b/tests/guests/guests.xml.in @@ -279,4 +279,20 @@ </devices> </domain> + <domain type='test'> + <name>windows-overlay</name> + <memory>1048576</memory> + <os> + <type>hvm</type> + <boot dev='hd'/> + </os> + <devices> + <disk type='file' device='disk'> + <driver name='qemu' type='qcow2'/> + <source file='@abs_builddir@/windows-overlay.qcow2'/> + <target dev='vda' bus='virtio'/> + </disk> + </devices> + </domain> + </node> diff --git a/v2v/Makefile.am b/v2v/Makefile.am index 06da002..dae063c 100644 --- a/v2v/Makefile.am +++ b/v2v/Makefile.am @@ -232,6 +232,7 @@ TESTS += \ test-v2v-cdrom.sh \ test-v2v-i-ova.sh \ test-v2v-i-disk.sh \ + test-v2v-in-place.sh \ test-v2v-machine-readable.sh \ test-v2v-networks-and-bridges.sh \ test-v2v-no-copy.sh \ diff --git a/v2v/cmdline.ml b/v2v/cmdline.ml index eaf57dc..2a3224a 100644 --- a/v2v/cmdline.ml +++ b/v2v/cmdline.ml @@ -37,6 +37,7 @@ let parse_cmdline () = let output_format = ref "" in let output_name = ref "" in let output_storage = ref "" in + let in_place = ref false in let password_file = ref "" in let print_source = ref false in let qemu_boot = ref false in @@ -160,6 +161,7 @@ let parse_cmdline () = "-of", Arg.Set_string output_format, "raw|qcow2 " ^ s_"Set output format"; "-on", Arg.Set_string output_name, "name " ^ s_"Rename guest when converting"; "-os", Arg.Set_string output_storage, "storage " ^ s_"Set output storage location"; + "--in-place", Arg.Set in_place, " " ^ s_"Only tune the guest in the input VM"; "--password-file", Arg.Set_string password_file, "file " ^ s_"Use password from file"; "--print-source", Arg.Set print_source, " " ^ s_"Print source and stop"; "--qemu-boot", Arg.Set qemu_boot, " " ^ s_"Boot in qemu (-o qemu only)"; @@ -226,6 +228,7 @@ read the man page virt-v2v(1). let output_mode = !output_mode in let output_name = match !output_name with "" -> None | s -> Some s in let output_storage = !output_storage in + let in_place = !in_place in let password_file = match !password_file with "" -> None | s -> Some s in let print_source = !print_source in let qemu_boot = !qemu_boot in @@ -305,6 +308,8 @@ read the man page virt-v2v(1). Input_ova.input_ova filename in (* Parse the output mode. *) + if output_mode <> `Not_set && in_place then + error (f_"-o and --in-place cannot be used at the same time"); let output = match output_mode with | `Glance -> @@ -386,5 +391,5 @@ read the man page virt-v2v(1). input, output, debug_gc, debug_overlays, do_copy, network_map, no_trim, - output_alloc, output_format, output_name, + output_alloc, output_format, output_name, in_place, print_source, root_choice diff --git a/v2v/test-v2v-in-place.sh b/v2v/test-v2v-in-place.sh new file mode 100755 index 0000000..c19707e --- /dev/null +++ b/v2v/test-v2v-in-place.sh @@ -0,0 +1,81 @@ +#!/bin/bash - +# libguestfs virt-v2v test script +# Copyright (C) 2014 Red Hat Inc. +# Copyright (C) 2015 Parallels IP Holdings GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Test --in-place. + +unset CDPATH +export LANG=C +set -e + +if [ -n "$SKIP_TEST_V2V_IN_PLACE_SH" ]; then + echo "$0: test skipped because environment variable is set" + exit 77 +fi + +if [ "$(guestfish get-backend)" = "uml" ]; then + echo "$0: test skipped because UML backend does not support network" + exit 77 +fi + +# You shouldn't be running the tests as root anyway, but in this case +# it's especially bad because we don't want to start creating guests +# or storage pools in the system namespace. +if [ "$(id -u)" -eq 0 ]; then + echo "$0: test skipped because you're running tests as root. Don't do that!" + exit 77 +fi + +guests_dir="$(cd $(dirname $0)/../tests/guests && pwd)" +libvirt_uri="test://$guests_dir/guests.xml" + +f="$guests_dir/windows.img" +if ! test -f $f || ! test -s $f; then + echo "$0: test skipped because phony Windows image was not created" + exit 77 +fi + +virt_tools_data_dir=${VIRT_TOOLS_DATA_DIR:-/usr/share/virt-tools} +if ! test -r $virt_tools_data_dir/rhsrvany.exe; then + echo "$0: test skipped because rhsrvany.exe is not installed" + exit 77 +fi + +fo="$guests_dir/windows-overlay.qcow2" +rm -f $fo +qemu-img create -f qcow2 -b $f -o compat=1.1,backing_fmt=raw $fo +md5="$(md5sum $f)" + +$VG virt-v2v --debug-gc \ + -i libvirt -ic "$libvirt_uri" windows-overlay \ + --in-place + +# Test some aspects of the target disk image. +guestfish --ro -a $fo -i <<EOF + is-dir "/Program Files/Red Hat/Firstboot" + is-file "/Program Files/Red Hat/Firstboot/firstboot.bat" + is-dir "/Program Files/Red Hat/Firstboot/scripts" + is-dir "/Windows/Drivers/VirtIO" +EOF + + +# Test the base image remained untouched +test "$md5" = "$(md5sum $f)" + +# Clean up. +rm -r $fo diff --git a/v2v/v2v.ml b/v2v/v2v.ml index 242f129..b744056 100644 --- a/v2v/v2v.ml +++ b/v2v/v2v.ml @@ -47,7 +47,8 @@ let rec main () = (* Handle the command line. *) let input, output, debug_gc, debug_overlays, do_copy, network_map, no_trim, - output_alloc, output_format, output_name, print_source, root_choice = + output_alloc, output_format, output_name, in_place, print_source, + root_choice = Cmdline.parse_cmdline () in (* Print the version, easier than asking users to tell us. *) @@ -117,52 +118,70 @@ let rec main () = ) nics in { source with s_nics = nics } in - (* Create a qcow2 v3 overlay to protect the source image(s). There - * is a specific reason to use the newer qcow2 variant: Because the - * L2 table can store zero clusters efficiently, and because - * discarded blocks are stored as zero clusters, this should allow us - * to fstrim/blkdiscard and avoid copying significant parts of the - * data over the wire. - *) - message (f_"Creating an overlay to protect the source from being modified"); let overlay_dir = (new Guestfs.guestfs ())#get_cachedir () in - let overlays = - List.map ( - fun ({ s_qemu_uri = qemu_uri; s_format = format } as source) -> - let overlay_file = - Filename.temp_file ~temp_dir:overlay_dir "v2vovl" ".qcow2" in - unlink_on_exit overlay_file; - - let options = - "compat=1.1" ^ - (match format with None -> "" - | Some fmt -> ",backing_fmt=" ^ fmt) in - let cmd = - sprintf "qemu-img create -q -f qcow2 -b %s -o %s %s" - (quote qemu_uri) (quote options) overlay_file in - if verbose () then printf "%s\n%!" cmd; - if Sys.command cmd <> 0 then - error (f_"qemu-img command failed, see earlier errors"); - - (* Sanity check created overlay (see below). *) - if not ((new G.guestfs ())#disk_has_backing_file overlay_file) then - error (f_"internal error: qemu-img did not create overlay with backing file"); - - overlay_file, source - ) source.s_disks in + let overlays = ( + if not in_place then ( + (* Create a qcow2 v3 overlay to protect the source image(s). There + * is a specific reason to use the newer qcow2 variant: Because the + * L2 table can store zero clusters efficiently, and because + * discarded blocks are stored as zero clusters, this should allow us + * to fstrim/blkdiscard and avoid copying significant parts of the + * data over the wire. + *) + message (f_"Creating an overlay to protect the source from being modified"); + List.map ( + fun ({ s_qemu_uri = qemu_uri; s_format = format } as source) -> + let overlay_file = + Filename.temp_file ~temp_dir:overlay_dir "v2vovl" ".qcow2" in + unlink_on_exit overlay_file; + + let options = + "compat=1.1" ^ + (match format with None -> "" + | Some fmt -> ",backing_fmt=" ^ fmt) in + let cmd = + sprintf "qemu-img create -q -f qcow2 -b %s -o %s %s" + (quote qemu_uri) (quote options) overlay_file in + if verbose () then printf "%s\n%!" cmd; + if Sys.command cmd <> 0 then + error (f_"qemu-img command failed, see earlier errors"); + + (* Sanity check created overlay (see below). *) + if not ((new G.guestfs ())#disk_has_backing_file overlay_file) then + error (f_"internal error: qemu-img did not create overlay with backing file"); + + overlay_file, source + ) source.s_disks + ) else [] + ) in (* Open the guestfs handle. *) - message (f_"Opening the overlay"); + if in_place then + message (f_"Opening the guest disks") + else + message (f_"Opening the overlay"); let g = new G.guestfs () in if trace () then g#set_trace true; if verbose () then g#set_verbose true; g#set_network true; - List.iter ( - fun (overlay_file, _) -> - g#add_drive_opts overlay_file - ~format:"qcow2" ~cachemode:"unsafe" ~discard:"besteffort" - ~copyonread:true - ) overlays; + if in_place then ( + List.iter ( + fun ({s_qemu_uri = qemu_uri; s_format = format}) -> + match format with + | None -> + g#add_drive_opts qemu_uri ~cachemode:"unsafe" ~discard:"besteffort" + | Some fmt -> + g#add_drive_opts qemu_uri ~format:fmt ~cachemode:"unsafe" + ~discard:"besteffort" + ) source.s_disks + ) else ( + List.iter ( + fun (overlay_file, _) -> + g#add_drive_opts overlay_file + ~format:"qcow2" ~cachemode:"unsafe" ~discard:"besteffort" + ~copyonread:true + ) overlays + ); g#launch (); @@ -294,6 +313,9 @@ let rec main () = g#shutdown (); g#close (); + if in_place then + exit 0; + (* Does the guest require UEFI on the target? *) message (f_"Checking if the guest needs BIOS or UEFI to boot"); let target_firmware = diff --git a/v2v/virt-v2v.pod b/v2v/virt-v2v.pod index ef2dee1..eda1cf7 100644 --- a/v2v/virt-v2v.pod +++ b/v2v/virt-v2v.pod @@ -9,6 +9,8 @@ virt-v2v - Convert a guest to use KVM virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi vmware_guest \ -o rhev -os rhev.nfs:/export_domain --network rhevm + virt-v2v -ic qemu:///system qemu_guest --in-place + virt-v2v -i libvirtxml guest-domain.xml -o local -os /var/tmp virt-v2v -i disk disk.img -o local -os /var/tmp @@ -75,6 +77,9 @@ booting the guest directly in qemu (mainly for testing). I<-o rhev> is used to write to a RHEV-M / oVirt target. I<-o vdsm> is only used when virt-v2v runs under VDSM control. +I<--in-place> instructs virt-v2v to customize the guest OS in the input +virtual machine, instead of creating a new VM in the target hypervisor. + =head1 EXAMPLES =head2 Convert from VMware vCenter server to local libvirt @@ -518,6 +523,18 @@ C<root>. You will get an error if virt-v2v is unable to mount/write to the Export Storage Domain. +=item B<--in-place> + +Do not create an output virtual machine in the target hypervisor. +Instead, adjust the guest OS in the source VM to run in the input +hypervisor. + +This mode is meant for integration with other toolsets, which take the +responsibility of converting the VM configuration, providing for +rollback in case of errors, transforming the storage, etc. + +Conflicts with all I<-o *> options. + =item B<--password-file> file Instead of asking for password(s) interactively, pass the password -- 2.4.3

3 74

ANNOUNCE: libguestfs & virt-v2v & virt-p2v RHEL 7.2 preview packages available
by Richard W.M. Jones 24 Sep '15

24 Sep '15

[The subject line isn't strictly correct, as virt-p2v packages aren't available right now, but will be shortly] If you are running RHEL 7.1 then you may be interested in trying out the newer RHEL 7.2 preview versions of: - libguestfs - virt-v2v (Technical Preview in RHEL 7.1, Supported in RHEL 7.2) - virt-p2v (Technical Preview in RHEL 7.2) To install the preview repository, copy the attached file into /etc/yum.repos.d/ Then use this command to update virt-v2v: yum install virt-v2v libguestfs-tools-c libguestfs-xfs To convert Windows guests you'll also need libguestfs-winsupport which is finally available in RHN (V2VWIN channel) or you can get it from http://people.redhat.com/~rjones/libguestfs-winsupport/ Please note these packages are *not* supported by Red Hat. If you find bugs, please file them using the link below: https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20… To downgrade back to the RHEL 7.1 supported packages, delete /etc/yum.repos.d/libguestfs-RHEL-7.2-preview.repo , uninstall libguestfs, then reinstall libguestfs from RHN. I have not tested these packages on CentOS 7.1, but they will probably work there too, and if they don't then let me know. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

1 2

missing btrfs subvol support
by Olaf Hering 10 Sep '15

10 Sep '15

Is btrfs subvol support failing just for me? Looks like nothing adds the required '@/' string. virt-ls uses the first variant of the command: ><rescue> mount -vo subvol=var/spool,ro /dev/sda2 /sysroot/ [ 113.852047] BTRFS info (device sda2): disk space caching is enabled [ 113.852869] BTRFS: has skinny extents mount: mount(2) failed: No such file or directory ><rescue> mount -vo subvol=@var/spool,ro /dev/sda2 /sysroot/ [ 116.082642] BTRFS info (device sda2): disk space caching is enabled [ 116.083476] BTRFS: has skinny extents mount: mount(2) failed: No such file or directory ><rescue> mount -vo subvol=@/var/spool,ro /dev/sda2 /sysroot/ [ 123.995343] BTRFS info (device sda2): disk space caching is enabled [ 123.996238] BTRFS: has skinny extents mount: /dev/sda2 mounted on /sysroot. ><rescue> exit fstab is: UUID=d538a81f-9778-424d-96cc-e48dd2d4a323 swap swap defaults 0 0 UUID=65b72f12-eccb-4cf2-a4f4-9c4bb462456f / btrfs defaults 0 0 UUID=65b72f12-eccb-4cf2-a4f4-9c4bb462456f /var/spool btrfs subvol=@/var/spool 0 0 Olaf

4 10

[PATCH v2 00/14] tests: Introduce test harness for running tests.
by Richard W.M. Jones 31 Jul '15

31 Jul '15

This is a more complete patch to add the test harness. The only parts missing now are the language bindings (except OCaml). The language bindings need a bit more thought. At the moment most language binding tests are done through some sort of shell script like perl/run-perl-tests which either runs each test itself or uses some language-specific machinary to run each test. The problem with that is the test harness should be running individual tests, otherwise we cannot parallelize them, nor run fast-only tests. That won't work however until the test harness knows how to run language tests, which it doesn't at the moment (except for OCaml). In any case, this is now ready for review, and maybe inclusion upstream. Rich.

1 14

Re: [Libguestfs] \n didn't lead to a new line while using remote model
by Richard W.M. Jones 30 Jul '15

30 Jul '15

On Wed, Jul 29, 2015 at 04:18:02PM +0800, Yu Liu wrote: [...] > Let's make an example: > > #guestfish -a disk.img > >run > >mount /dev/sda1 / > >write-append /a.txt "Hello\n" > >write-append /a.txt "World\n" > >cat /a.txt > Hello > World > > >quit > > Another try: > eval `guestfish --listen` > guestfish --remote add disk.img > guestfish --remote run > guestfish --remote mount /dev/sda1 / > guestfish --remote write-append /a.txt "Hello\n" > guestfish --remote write-append /a.txt "World\n" > guestfish --remote cat /a.txt > Hello\nWorld\n > > # > libguestfs-1.20.11-11.el6.x86_64 The problem is that \n is handled "specially" by guestfish when it is reading the ><fs> command line, but not when it is parsing commands sent via --remote. The way to do this is: $ guestfish --remote write /a.txt "hello world" $ guestfish --remote cat /a.txt hello world Note that you have to actually press the [Return] key after "hello Probably a better plan is to use something like python remoting: https://rwmj.wordpress.com/2013/05/07/using-libguestfs-remotely-with-python… which will be more predictable. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

2 5

using virt-v2v
by Arsène Gschwind 30 Jul '15

30 Jul '15

Hi, I'm trying to migrate some VMs from KVM to oVirt using virt-v2v and I cannot get it to work. Where do I have to install virt-v2v on the oVirt management host? I've setup a RHEL 7.1 VM with the libguestfs preview for RHEL 7.2, will this work that way. My VM are located on FC LUNs on the KVM server, I'm able to access libvirt using qemu+ssh but the it returns an error that it cannot find the device where the VM is located on. What will be the best/working strategy for doing such a migration. Thanks for any hint/help. Rgds, Arsène

2 2

restrict access to host from guestfish
by Raghu 29 Jul '15

29 Jul '15

Hi Richard, guestfish shell has an ability to execute commands on the host such as !mkdir local tgz-out /remote local/remote-data.tar.gz What is the best way to restrict access to host from guestfish ? For instance, - Allow readonly access to host.. i.e., !ls is allowed but dont allow !rm or !mkdir - commands such as tgz-out, or copy-out should be able to access just /tmp, but nothing else in host filesystem Appreciate your guidance on this, Thanks Raghu

2 1

Redhat v2v
by Philip KP LI 29 Jul '15

29 Jul '15

Hi, I am looking for the package, virt-v2v, for Redhat 7 to convert virtual machines under Xen in Redhat 5 to KVM. But, can't find it from the 7 and neither from the public websites. Why wasn’t virt-v2v included with 7 from the get-go? And, pls also advise for the downloading sites if possible. Regards, Philip Li ____________________________________________________________________________________________________________ This e-mail message (together with any attachments) is for the designated recipient only. It may contain information that is privileged for the designated recipient. If you are not the intended recipient, you are hereby notified that any use, retention, disclosure, copying, printing, forwarding or dissemination of the message is strictly prohibited. If you have received the message in error, please erase all copies of the message (including attachments) from your system and notify the sender immediately. ____________________________________________________________________________________________________________

2 1