October 2016 - Libguestfs - libguestfs List Archives

[PATCH v2 0/2] Improve OVA manifest parsing
by Tomáš Golembiovský 05 Oct '16

05 Oct '16

This series fixes and enhances parsing of the OVA manifest file. The changes are: - Added mandatory space to the regexp - Process all lines in the file, not just one - Warn on improperly formated lines - Support SHA256 hashes v1 -> v2: rebased on to master to make use of the Checksums module Tomáš Golembiovský (2): v2v: ova: fix checking of the manifest file v2v: ova: support SHA256 hashes in manifest v2v/input_ova.ml | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) -- 2.10.0

2 4

Re: [Libguestfs] [Arm-dev] Disk Image Creation Suggestions
by Richard W.M. Jones 05 Oct '16

05 Oct '16

On Mon, Oct 03, 2016 at 02:22:04PM -0400, Christopher Covington wrote: > Hi, > > I'm running Anaconda regularly and I plan on continuing to do so, but > there are certain circumstances where I'd like to lock down all the > package versions and provision many similar systems as quickly as > possible. So I'm looking for an automated and reliable mechanism for > generating disk images for use on "bare metal" or physical (not > virtualized) installs. The images will probably be copied to SATA*. > > How are the disk images currently generated? Does Jim manually launch > an install with the documented [1] kickstart file and then move the > storage medium to another machine and run dd? Or is it more automated? > > 1. http://mirror.centos.org/altarch/7/isos/aarch64/ReadMe.txt > > Looking through the archives I see Richard announcing virt-builder and > virt-install availability for AArch64. It looks very useful for this > sort of task. Are there any reasons to not use its output on a > non-virtualized, physical instance? ("V2P"?) Yes, you can provision baremetal machines using virt-builder. I have done this and it works just fine. Plug in the SATA disk, and simply do: virt-builder --arch aarch64 centos-7.2 -o /dev/sdX Note that you will probably need to use same arch for the provisioning(? host? whatever the terminology is) machine as for the target, otherwise useful options like --install won't work. Rich. > Then there's livemedia-creator [2]. Would it be best to use that to > wrap the virt-install invocation? > > 2. http://lorax.readthedocs.io/en/latest/livemedia-creator.html > > If anyone has experience and guidance with the above tools or > alternatives, especially on AArch64, it would be appreciated. The > documentation looks very good for these tools but some aspects > like authoring kickstart files which are compatible with > lorax/livemedia-creator seem particularly tricky. > > Thanks, > Cov > > * Not essential, but having the option to do some kind of read-only > Network Block Device (NBD) root filesystem which many systems could > share, with a local, writeable, essentially throw-away overlay might > be handy. > > -- > Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm > Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code > Aurora Forum, a Linux Foundation Collaborative Project. > _______________________________________________ > Arm-dev mailing list > Arm-dev(a)centos.org > https://lists.centos.org/mailman/listinfo/arm-dev -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

1 0

Re: [Libguestfs] [Bug 1381502] virt-sparsify doesn't remove content of deleted files on XFS
by Richard W.M. Jones 04 Oct '16

04 Oct '16

On Tue, Oct 04, 2016 at 02:36:53PM +0000, MUCHOWSKI, Piotr (EXT - PL) wrote: > Good afternoon Richard, > > Just one question about "--scrub" option. Can the "scrub mode" be > enabled globally (not just for single FILE after this option)? > Preferably for all sysprep items where some files are deleted. For > example logfiles rh-subscription-manager rhn-systemid and so on. No, it doesn't work this way. You have to scrub individual files. Note this uses the 'scrub(1)' command. > And finally, is any requirement the CentOS/RedHat images must be > processed by virt-sysprep running on CentOS host and not on Ubuntu > (which has different kernel). Because what I observed, these tools > get the host kernel from /boot directory. I didn't found how to > provide arbitrary kernel file or use the kernel built in processed > image. You can use a different kernel by setting SUPERMIN_* environment variables: http://libguestfs.org/supermin.1.html#USING-A-CUSTOM-KERNEL-AND-KERNEL-MODU… http://libguestfs.org/guestfs-faq.1.html#how-can-i-compile-and-install-libg… In theory you could extract the kernel from the disk image using 'virt-get-kernel'. It doesn't currently let you extract the modules from the same disk image [that would be a nice feature], but you could do that with a trivial bit of guestfish scripting. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

1 0

[PATCH] v2v: linux: try to fix removal of VMware tools
by Pino Toscano 04 Oct '16

04 Oct '16

Try to improve the way packages of VMware tools are removed from YUM-based guests: - when filtering the package itself from its providers, do a stricter check so either the provide is the unversioned package, or it is exactly its own name - if the package has no other providers, then going further will cause the invocation of 'yum install' with no packages, and thus the package itself will not be added to the list of packages to be removed; to overcome this issue, just mark the package as "to be removed" in that case Related to: RHBZ#1155150 --- v2v/convert_linux.ml | 56 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 34 insertions(+), 22 deletions(-) diff --git a/v2v/convert_linux.ml b/v2v/convert_linux.ml index 6b4197d..8771869 100644 --- a/v2v/convert_linux.ml +++ b/v2v/convert_linux.ml @@ -252,29 +252,41 @@ let rec convert ~keep_serial_console (g : G.guestfs) inspect source rcaps = (* The packages provide themselves, filter this out. *) let provides = - List.filter (fun s -> String.find s library = -1) provides in - - (* Trim whitespace. *) - let rex = Str.regexp "^[ \t]*\$[^ \t]+\$[ \t]*$" in - let provides = List.map (Str.replace_first rex "\\1") provides in - - (* Install the dependencies with yum. Use yum explicitly - * because we don't have package names and local install is - * impractical. + List.filter ( + fun s -> + not (library = s || String.is_prefix s (library ^ " = ")) + ) provides in + + (* If the package provides something other than itself, then + * proceed installing the replacements; in the other case, + * just mark the package for removal, as it means no other + * package can depend on something provided. *) - let cmd = ["yum"; "-q"; "resolvedep"] @ provides in - let cmd = Array.of_list cmd in - let replacements = g#command_lines cmd in - let replacements = Array.to_list replacements in - - let cmd = [ "yum"; "install"; "-y" ] @ replacements in - let cmd = Array.of_list cmd in - (try - ignore (g#command cmd); - push_front library remove - with G.Error msg -> - eprintf "%s: could not install replacement for %s. Error was: %s. %s was not removed.\n" - prog library msg library + if provides <> [] then ( + (* Trim whitespace. *) + let rex = Str.regexp "^[ \t]*\$[^ \t]+\$[ \t]*$" in + let provides = List.map (Str.replace_first rex "\\1") provides in + + (* Install the dependencies with yum. Use yum explicitly + * because we don't have package names and local install is + * impractical. + *) + let cmd = ["yum"; "-q"; "resolvedep"] @ provides in + let cmd = Array.of_list cmd in + let replacements = g#command_lines cmd in + let replacements = Array.to_list replacements in + + let cmd = [ "yum"; "install"; "-y" ] @ replacements in + let cmd = Array.of_list cmd in + (try + ignore (g#command cmd); + push_front library remove + with G.Error msg -> + eprintf "%s: could not install replacement for %s. Error was: %s. %s was not removed.\n" + prog library msg library + ); + ) else ( + push_front library remove; ); ) libraries ) -- 2.7.4

2 1

[PATCH v2 0/2] New tool: virt-tail.
by Richard W.M. Jones 03 Oct '16

03 Oct '16

Nothing new in the virt-tail command itself, but the second commit includes a simple test. Rich.

2 4

[PATCH] nbdkit: flags are 32 bits for oldstyle connections
by Carl-Daniel Hailfinger 03 Oct '16

03 Oct '16

# HG changeset patch # User Carl-Daniel Hailfinger <Carl-Daniel.Hailfinger(a)inf.h-brs.de> # Date 1474903734 -7200 # Mon Sep 26 17:28:54 2016 +0200 # Node ID dbd1ea0a401cedcfa426097a289d852971b14f1e # Parent d7d5078d08c711032837dcac79a4450226ec2ce5 nbdkit: Fix flags in old-style connection diff -r d7d5078d08c7 -r dbd1ea0a401c src/connections.c --- a/src/connections.c Sun Sep 25 05:04:02 2016 +0200 +++ b/src/connections.c Mon Sep 26 17:28:54 2016 +0200 @@ -155,7 +155,7 @@ struct old_handshake handshake; int64_t r; uint64_t exportsize; - uint16_t gflags, eflags; + uint32_t flags; int fl; r = plugin_get_size (conn); @@ -169,14 +169,13 @@ exportsize = (uint64_t) r; conn->exportsize = exportsize; - gflags = 0; - eflags = NBD_FLAG_HAS_FLAGS; + flags = NBD_FLAG_HAS_FLAGS; fl = plugin_can_write (conn); if (fl == -1) return -1; if (readonly || !fl) { - eflags |= NBD_FLAG_READ_ONLY; + flags |= NBD_FLAG_READ_ONLY; conn->readonly = 1; } @@ -184,7 +183,7 @@ if (fl == -1) return -1; if (fl) { - eflags |= NBD_FLAG_SEND_FLUSH | NBD_FLAG_SEND_FUA; + flags |= NBD_FLAG_SEND_FLUSH | NBD_FLAG_SEND_FUA; conn->can_flush = 1; } @@ -192,7 +191,7 @@ if (fl == -1) return -1; if (fl) { - eflags |= NBD_FLAG_ROTATIONAL; + flags |= NBD_FLAG_ROTATIONAL; conn->is_rotational = 1; } @@ -200,7 +199,7 @@ if (fl == -1) return -1; if (fl) { - eflags |= NBD_FLAG_SEND_TRIM; + flags |= NBD_FLAG_SEND_TRIM; conn->can_trim = 1; } @@ -208,19 +207,17 @@ if (fl == -1) return -1; if (fl) { - eflags |= NBD_FLAG_SEND_MARKNOREMANENCE; + flags |= NBD_FLAG_SEND_MARKNOREMANENCE; conn->can_marknoremanence = 1; } - debug ("oldstyle negotiation: flags: global 0x%x export 0x%x", - gflags, eflags); + debug ("oldstyle negotiation: server flags: 0x%x", flags); memset (&handshake, 0, sizeof handshake); memcpy (handshake.nbdmagic, "NBDMAGIC", 8); handshake.version = htobe64 (OLD_VERSION); handshake.exportsize = htobe64 (exportsize); - handshake.gflags = htobe16 (gflags); - handshake.eflags = htobe16 (eflags); + handshake.flags = htobe32 (flags); if (xwrite (conn->sockout, &handshake, sizeof handshake) == -1) { nbdkit_error ("write: %m"); diff -r d7d5078d08c7 -r dbd1ea0a401c src/protocol.h --- a/src/protocol.h Sun Sep 25 05:04:02 2016 +0200 +++ b/src/protocol.h Mon Sep 26 17:28:54 2016 +0200 @@ -41,8 +41,7 @@ char nbdmagic[8]; /* "NBDMAGIC" */ uint64_t version; /* OLD_VERSION, in network byte order */ uint64_t exportsize; /* in network byte order */ - uint16_t gflags; /* global flags, in network byte order */ - uint16_t eflags; /* per-export flags, in network byte order */ + uint32_t flags; /* flags, in network byte order */ char zeroes[124]; /* must be sent as zero bytes */ } __attribute__((packed));

2 1

Extracting files from OVA is bad
by Tomáš Golembiovský 03 Oct '16

03 Oct '16

Hi, recently we (oVirt) have started discussing whether the way virt-v2v handles import from OVA files is good. And I would be interested in ideas how it can be improved. It is likely somebody already gave some thought to this problem. TL;DR: Extracting the OVA before import is a problem for large VMs (in sizes of TBs). Can we change something to prevent the extraction and work directly over OVA? What we consider a huge shortcoming is the fact that whole OVA is extracted prior to the import into a temporary directory and processed afterwards. Under normal situation user can have up to three copies of the VM on his drive at the end of import: * original OVA, * temporary extracted files (will be deleted when virt-v2v terminates, * converted VM. This is not a good idea for large VMs that have hunderds of GBs or even TBs in size. The requirements on the necessary storage space can be lessened with proper partitioning. I.e. source OVA and converted VM don't end up on the same drive and TMPDIR is set to put even temporary files somewhere else. But this is not a general solution. And sometimes the necessary space may not be available at all. The question is how to change the import path so that virt-v2v doesn't have to extract the OVA. I can see the following solutions: 1) Solve it virt-v2v: create a layer for directly accessing the files in the archive. 2) Solve it in QEMU: create backing method that would allow creating qemu disk backed by the archive. 3) Solve it on oVirt side: use some FUSE-based tool to provide access to the archive and pass the OVA to virt-v2v not as a file but as directory. Does anyone have any other ideas or suggestions? Best regards, Tomas Golembiovsky -- Tomáš Golembiovský <tgolembi(a)redhat.com>

4 6

[PATCH] appliance: Add ExFAT packages.
by Richard W.M. Jones 03 Oct '16

03 Oct '16

Thanks: Miles Wolbe --- appliance/packagelist.in | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/appliance/packagelist.in b/appliance/packagelist.in index d2a065f..77bd464 100644 --- a/appliance/packagelist.in +++ b/appliance/packagelist.in @@ -288,5 +288,10 @@ kpartx dnl (virt-dib) tools optionally used for elements debootstrap +dnl exFAT is not usually available in free software repos +exfat-fuse +exfat-utils +fuse-exfat + dnl Define this by doing: ./configure --with-extra-packages="..." EXTRA_PACKAGES -- 2.9.3

2 1

[PATCH v2 0/3] v2v: further bits of Debian/Ubuntu guests supports
by Pino Toscano 03 Oct '16

03 Oct '16

Hi, this series adds a couple bits more in v2v to convert Debian/Ubuntu (and derived) guests. The series does not complete the support (see known issues below), but all the patches here should be fit for review and inclusion. The series does not enable the conversion, yet. Known issues: * currently tested with simple local guest images, hence needs testing with real guests on libvirt/VMware/Xen/VirtualBox/etc (which have various tools to be removed/tweaked) * is tweaking the "enterprise-linux" module the correct/wanted way? should it be renamed to just "linux" then? * surely something else I'm missing Changes in v2: - dropped former patch #1, merged - reorder former patch #3 as #1 - add kernel config file detection as patch #2 - adapt patch #3 to use the found config file, if available Thanks, Pino Toscano (3): v2v: bootloaders: improve detection of Grub2 default method v2v: linux kernels: detect config file v2v: linux: check also kernel config for modules v2v/linux_bootloaders.ml | 61 ++++++++++++++++++++++++++++++++++-------------- v2v/linux_kernels.ml | 31 ++++++++++++++++++++++-- v2v/linux_kernels.mli | 1 + 3 files changed, 74 insertions(+), 19 deletions(-) -- 2.7.4

2 4

[PATCH 0/3] v2v: further bits of Debian/Ubuntu guests supports
by Pino Toscano 03 Oct '16

03 Oct '16

Hi, this series adds a couple bits more in v2v to convert Debian/Ubuntu (and derived) guests. The series does not complete the support (see known issues below), but all the patches here should be fit for review and inclusion. The series does not enable the conversion, yet. Known issues: * currently tested with simple local guest images, hence needs testing with real guests on libvirt/VMware/Xen/VirtualBox/etc (which have various tools to be removed/tweaked) * is tweaking the "enterprise-linux" module the correct/wanted way? should it be renamed to just "linux" then? * surely something else I'm missing Thanks, Pino Toscano (3): v2v: linux kernels: set type of 'g' v2v: linux: check also kernel config for modules v2v: bootloaders: improve detection of Grub2 default method v2v/linux_bootloaders.ml | 61 ++++++++++++++++++++++++++++++++++-------------- v2v/linux_kernels.ml | 22 +++++++++++++++-- 2 files changed, 64 insertions(+), 19 deletions(-) -- 2.7.4

3 9