March 2019 - Libguestfs - libguestfs List Archives

[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin 06 Feb '20

06 Feb '20

This feature allows you to use different image formats for the fixed appliance. The raw format is used by default. Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com> --- lib/launch-direct.c | 2 ++ lib/launch-libvirt.c | 19 ++++++++++++------- m4/guestfs_appliance.m4 | 11 +++++++++++ 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index 0be662e25..b9b54857a 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg) append_list ("id=appliance"); append_list ("cache=unsafe"); append_list ("if=none"); +#ifndef APPLIANCE_FMT_AUTO append_list ("format=raw"); +#endif } end_list (); start_list ("-device") { append_list ("scsi-hd"); diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c index 4adb2cfb3..030ea6911 100644 --- a/lib/launch-libvirt.c +++ b/lib/launch-libvirt.c @@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv) /** * Create a qcow2 format overlay, with the given C<backing_drive> - * (file). The C<format> parameter, which must be non-NULL, is the - * backing file format. This is used to create the appliance overlay, - * and also for read-only drives. + * (file). The C<format> parameter is the backing file format. + * The C<format> parameter can be NULL, in this case the backing + * format will be determined automatically. This is used to create + * the appliance overlay, and also for read-only drives. */ static char * make_qcow2_overlay (guestfs_h *g, const char *backing_drive, @@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, char *overlay; struct guestfs_disk_create_argv optargs; - assert (format != NULL); - if (guestfs_int_lazy_make_tmpdir (g) == -1) return NULL; @@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK; optargs.backingfile = backing_drive; - optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; - optargs.backingformat = format; + if (format) { + optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; + optargs.backingformat = format; + } if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) { free (overlay); @@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri) /* Note that appliance can be NULL if using the old-style appliance. */ if (appliance) { +#ifdef APPLIANCE_FMT_AUTO + params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL); +#else params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw"); +#endif if (!params.appliance_overlay) goto cleanup; } diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4 index 81c43879f..4e1ec8135 100644 --- a/m4/guestfs_appliance.m4 +++ b/m4/guestfs_appliance.m4 @@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH]) AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"], [Define guestfs default path.]) + +AC_ARG_ENABLE([appliance-fmt-auto], + [AS_HELP_STRING([--enable-appliance-fmt-auto], + [enable autodetection of appliance image format @<:@default=no@:>@])], + [ENABLE_APPLIANCE_FMT_AUTO="$enableval"], + [ENABLE_APPLIANCE_FMT_AUTO=no]) + +if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then + AC_DEFINE([APPLIANCE_FMT_AUTO], [1], + [Define to 1 if enabled autodetection of appliance image format.]) +fi -- 2.13.0

2 4

1.39 proposal: Let's split up the libguestfs git repo and tarballs
by Richard W.M. Jones 29 Nov '19

29 Nov '19

My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test. The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace. My suggested split would be something like this: * libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website * 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby * virt-customize and related tools, we'd probably call this subproject "virt-builder". It would include virt-builder, virt-customize and virt-sysprep, since they share a lot of common code. * 1 project for each of the following items: small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg * I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them. * common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified. M4, PO, and tests would be split between the projects as appropriate. My proposal would be to do this incrementally, rather than all at once, moving the easier things out first. Thoughts? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

4 18

[PATCH] v2v: rhv-upload-plugin - improve wait logic after finalize (RHBZ#1680361)
by Daniel Erez 09 Aug '19

09 Aug '19

After invoking transfer_service.finalize, check operation status by examining ImageTransferPhase and DiskStatus. This is done instead of failing after a predefined timeout regardless the status. * not verified * Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1680361 --- v2v/rhv-upload-plugin.py | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py index 2a950c5ed..873c11ce1 100644 --- a/v2v/rhv-upload-plugin.py +++ b/v2v/rhv-upload-plugin.py @@ -523,14 +523,30 @@ def close(h): # waiting for the transfer object to cease to exist, which # falls through to the exception case and then we can # continue. - endt = time.time() + timeout + start = time.time() try: while True: time.sleep(1) - tmp = transfer_service.get() - if time.time() > endt: - raise RuntimeError("timed out waiting for transfer " - "to finalize") + transfer = transfer_service.get() + + if transfer is None: + disk_service = h['disk_service'] + disk = disk_service.get() + if disk.status == types.DiskStatus.OK: + continue + + if transfer.phase == types.ImageTransferPhase.FINISHED_SUCCESS: + debug("finalized after %s seconds", time.time() - start) + break + + if transfer.phase == types.ImageTransferPhase.FINALIZING_SUCCESS: + if time.time() > start + timeout: + raise RuntimeError("timed out waiting for transfer " + "to finalize") + continue + + raise RuntimeError("Unexpected transfer phase while finalizing " + "upload %r" % transfer.phase) except sdk.NotFoundError: pass --

3 9

[PATCH nbdkit 0/7] server: Implement NBD_FLAG_CAN_MULTI_CONN.
by Richard W.M. Jones 15 May '19

15 May '19

First thing to say is that I need to do a *lot* more testing on this, so this is just an early peek. In particular, although it passed ‘make check && make check-valgrind’ I have *not* tested it against a multi-conn-aware client such as the Linux kernel >= 4.9. This implements NBD_FLAG_CAN_MULTI_CONN, described in the protocol doc as: "NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates entirely without cache, or that the cache it uses is shared among all connections to the given device. In particular, if this flag is present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST be visible across all connections when the server sends its reply to that command to the client. In the absence of this flag, clients SHOULD NOT multiplex their commands over more than one connection to the export." This is necessary to support the Linux nbd client -C option. The only plugin which sets the flag so far is file. The ocaml, sh and nbd plugins allow the flag to be controlled or passed through. Notable also is that the blocksize filter has to filter out this flag, because I'm not convinced that the bounce buffer is safe. However I believe the other filters *are* safe (although not really certain about the fua filter, and the new cache filter is tricky too). My feeling is that we should set the flag unconditionally for all readonly connections, but I can think of nasty corner cases where it might not work. We should most probably set it in all plugins that are readonly (eg. nbdkit-pattern-plugin). Rich.

2 15

[PATCH nbdkit v5 FINAL 00/19] Implement extents.
by Richard W.M. Jones 27 Apr '19

27 Apr '19

This has already been pushed upstream. I am simply posting these here so we have a reference in the mailing list in case we find bugs later (as I'm sure we will - it's a complex patch series). Great thanks to Eric Blake for tireless review on this one. It also seems to have identified a few minor bugs in qemu along the way. Rich.

3 49

[PATCH nbdkit] Add support for writing plugins in Rust.
by Richard W.M. Jones 24 Apr '19

24 Apr '19

This adds very rough support for writing nbdkit plugins in Rust. This is not very idiomatic -- essentially we're handling the direct C calls from nbdkit in Rust. We have to use ‘unsafe’ in a few places because there's no way to tell the Rust code that nbdkit satisfies guarantees (eg. around thread safety, always returning leaked pointers back to the close function, always doing bounds checking). See the TODO file changes for the mising bits. TBH there's not much to review here. Rich.

2 4

[RFC PATCH] protocol: Add NBD_CMD_FLAG_FAST_ZERO
by Eric Blake 12 Apr '19

12 Apr '19

While it may be counterintuitive at first, the introduction of NBD_CMD_WRITE_ZEROES and NBD_CMD_BLOCK_STATUS has caused a performance regression in qemu [1], when copying a sparse file. When the destination file must contain the same contents as the source, but it is not known in advance whether the destination started life with all zero content, then there are cases where it is faster to request a bulk zero of the entire device followed by writing only the portions of the device that are to contain data, as that results in fewer I/O transactions overall. In fact, there are even situations where trimming the entire device prior to writing zeroes may be faster than bare write zero request [2]. However, if a bulk zero request ever falls back to the same speed as a normal write, a bulk pre-zeroing algorithm is actually a pessimization, as it ends up writing portions of the disk twice. [1] https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg06389.html [2] https://github.com/libguestfs/nbdkit/commit/407f8dde Hence, it is desirable to have a way for clients to specify that a particular write zero request is being attempted for a fast wipe, and get an immediate failure if the zero request would otherwise take the same time as a write. Conversely, if the client is not performing a pre-initialization pass, it is still more efficient in terms of networking traffic to send NBD_CMD_WRITE_ZERO requests where the server implements the fallback to the slower write, than it is for the client to have to perform the fallback to send NBD_CMD_WRITE with a zeroed buffer. Add a protocol flag and corresponding transmission advertisement flag to make it easier for clients to inform the server of their intent. If the server advertises NBD_FLAG_SEND_FAST_ZERO, then it promises two things: to perform a fallback to write when the client does not request NBD_CMD_FLAG_FAST_ZERO (so that the client benefits from the lower network overhead); and to fail quickly with ENOTSUP if the client requested the flag but the server cannot write zeroes more efficiently than a normal write (so that the client is not penalized with the time of writing data areas of the disk twice). Note that the semantics are chosen so that servers should advertise the new flag whether or not they have fast zeroing (that is, this is NOT the server advertising that it has fast zeroes, but rather advertising that the client can get feedback as needed on whether zeroing is fast). It is also intentional that the new advertisement includes a new errno value, ENOTSUP, with rules that this error should not be returned for any pre-existing behaviors, must not happen when the client does not request a fast zero, and must be returned quickly if the client requested fast zero but anything other than the error would not be fast; while leaving it possible for clients to distinguish other errors like EINVAL if alignment constraints are not met. Clients should not send the flag unless the server advertised support, but well-behaved servers should already be reporting EINVAL to unrecognized flags. If the server does not advertise the new feature, clients can safely fall back to assuming that writing zeroes is no faster than normal writes. Note that the Linux fallocate(2) interface may or may not be powerful enough to easily determine if zeroing will be efficient - in particular, FALLOC_FL_ZERO_RANGE in isolation does NOT give that insight; for block devices, it is known that ioctl(BLKZEROOUT) does NOT have a way for userspace to probe if it is efficient or slow. But with enough demand, the kernel may add another FALLOC_FL_ flag to use with FALLOC_FL_ZERO_RANGE, and/or appropriate ioctls with guaranteed ENOTSUP failures if a fast path cannot be taken. If a server cannot easily determine if write zeroes will be efficient, it is better off not advertising NBD_FLAG_SEND_FAST_ZERO. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- I will not push this without both: - a positive review (for example, we may decide that burning another NBD_FLAG_* is undesirable, and that we should instead have some sort of NBD_OPT_ handshake for determining when the server supports NBD_CMF_FLAG_FAST_ZERO) - a reference client and server implementation (probably both via qemu, since it was qemu that raised the problem in the first place) doc/proto.md | 44 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/doc/proto.md b/doc/proto.md index 8aaad96..1107766 100644 --- a/doc/proto.md +++ b/doc/proto.md @@ -1059,6 +1059,17 @@ The field has the following format: which support the command without advertising this bit, and conversely that this bit does not guarantee that the command will succeed or have an impact. +- bit 11, `NBD_FLAG_SEND_FAST_ZERO`: allow clients to detect whether + `NBD_CMD_WRITE_ZEROES` is efficient. The server MUST set this + transmission flag to 1 if the `NBD_CMD_WRITE_ZEROES` request + supports the `NBD_CMD_FLAG_FAST_ZERO` flag, and MUST set this + transmission flag to 0 if `NBD_FLAG_SEND_WRITE_ZEROES` is not + set. Servers SHOULD NOT set this transmission flag if there is no + quick way to determine whether a particular write zeroes request + will be efficient, but the lack of an efficient write zero + implementation SHOULD NOT prevent a server from setting this + flag. Clients MUST NOT set the `NBD_CMD_FLAG_FAST_ZERO` request flag + unless this transmission flag is set. Clients SHOULD ignore unknown flags. @@ -1636,6 +1647,12 @@ valid may depend on negotiation during the handshake phase. MUST NOT send metadata on more than one extent in the reply. Client implementors should note that using this flag on multiple contiguous requests is likely to be inefficient. +- bit 4, `NBD_CMD_FLAG_FAST_ZERO`; valid during + `NBD_CMD_WRITE_ZEROES`. If set, but the server cannot perform the + write zeroes any faster than it would for an equivalent + `NBD_CMD_WRITE`, then the server MUST fail quickly with an error of + `ENOTSUP`. The client MUST NOT set this unless the server advertised + `NBD_FLAG_SEND_FAST_ZERO`. ##### Structured reply flags @@ -2004,7 +2021,10 @@ The following request types exist: reached permanent storage, unless `NBD_CMD_FLAG_FUA` is in use. A client MUST NOT send a write zeroes request unless - `NBD_FLAG_SEND_WRITE_ZEROES` was set in the transmission flags field. + `NBD_FLAG_SEND_WRITE_ZEROES` was set in the transmission flags + field. Additionally, a client MUST NOT send the + `NBD_CMD_FLAG_FAST_ZERO` flag unless `NBD_FLAG_SEND_FAST_ZERO` was + set in the transimssion flags field. By default, the server MAY use trimming to zero out the area, even if it did not advertise `NBD_FLAG_SEND_TRIM`; but it MUST ensure @@ -2014,6 +2034,23 @@ The following request types exist: same area will not cause fragmentation or cause failure due to insufficient space. + If the server advertised `NBD_FLAG_SEND_FAST_ZERO` but + `NBD_CMD_FLAG_FAST_ZERO` is not set, then the server MUST NOT fail + with `ENOTSUP`, even if the operation is no faster than a + corresponding `NBD_CMD_WRITE`. Conversely, if + `NBD_CMD_FLAG_FAST_ZERO` is set, the server MUST fail quickly with + `ENOTSUP` unless the request can be serviced more efficiently than + a corresponding `NBD_CMD_WRITE`. The server's determination of + efficiency MAY depend on whether the request was suitably aligned, + on whether the `NBD_CMD_FLAG_NO_HOLE` flag was present, or even on + whether a previous `NBD_CMD_TRIM` had been performed on the + region. If the server did not advertise + `NBD_FLAG_SEND_FAST_ZERO`, then it SHOULD NOT fail with `ENOTSUP`, + regardless of the speed of servicing a request, and SHOULD fail + with `EINVAL` if the `NBD_CMD_FLAG_FAST_ZERO` flag was set. A + server MAY advertise `NBD_FLAG_SEND_FAST_ZERO` whether or not it + can perform efficient zeroing. + If an error occurs, the server MUST set the appropriate error code in the error field. @@ -2114,6 +2151,7 @@ The following error values are defined: * `EINVAL` (22), Invalid argument. * `ENOSPC` (28), No space left on device. * `EOVERFLOW` (75), Value too large. +* `ENOTSUP` (95), Operation not supported. * `ESHUTDOWN` (108), Server is in the process of being shut down. The server SHOULD return `ENOSPC` if it receives a write request @@ -2125,6 +2163,10 @@ request is not aligned to advertised minimum block sizes. Finally, it SHOULD return `EPERM` if it receives a write or trim request on a read-only export. +The server SHOULD NOT return `ENOTSUP` except as documented in +response to `NBD_CMD_WRITE_ZEROES` when `NBD_CMD_FLAG_FAST_ZERO` is +supported. + The server SHOULD return `EINVAL` if it receives an unknown command. The server SHOULD return `EINVAL` if it receives an unknown command flag. It -- 2.20.1

5 7

small question regarding ovirt
by Hetz Ben Hamo 03 Apr '19

03 Apr '19

Hi, Small questoin: when running the P2v hard disk image, and I want to convert a phyical machine to oVirt, to which machine do I connect when the GUI starts? one of the nodes or the hosted engine? (i'm not talking about the URL). Thanks

3 2

[PATCH v2 0/3] v2v: add -o json output mode
by Pino Toscano 02 Apr '19

02 Apr '19

This series adds a new output mode for virt-v2v, called -o json. It produces local files, just like -o local, although the metadata produced is a JSON file with data that v2v collected in the conversion process. This can be useful for converting to unsupported destinations, still based on QEMU/KVM. In addition to a simple different metadata, it offers a way to relocate the disks, with %{...}-like variables (only 3 added ATM, more can be added) to change their paths depending on data of the guest/disks. Changes from v1: - remove extra code for optional named parameters in pcre_tests.ml - allow %-escaping of variables in Var_expander Pino Toscano (3): common/mlpcre: add offset flag for PCRE.matches v2v: add Var_expander v2v: add -o json output mode .gitignore | 1 + common/mlpcre/PCRE.ml | 2 +- common/mlpcre/PCRE.mli | 5 +- common/mlpcre/pcre-c.c | 16 +- common/mlpcre/pcre_tests.ml | 11 +- v2v/Makefile.am | 36 +++- v2v/cmdline.ml | 29 +++ v2v/create_json.ml | 348 ++++++++++++++++++++++++++++++++++ v2v/create_json.mli | 29 +++ v2v/dummy.c | 2 + v2v/output_json.ml | 116 ++++++++++++ v2v/output_json.mli | 31 +++ v2v/var_expander.ml | 72 +++++++ v2v/var_expander.mli | 82 ++++++++ v2v/var_expander_tests.ml | 113 +++++++++++ v2v/virt-v2v-output-local.pod | 55 ++++++ v2v/virt-v2v.pod | 15 +- 17 files changed, 951 insertions(+), 12 deletions(-) create mode 100644 v2v/create_json.ml create mode 100644 v2v/create_json.mli create mode 100644 v2v/dummy.c create mode 100644 v2v/output_json.ml create mode 100644 v2v/output_json.mli create mode 100644 v2v/var_expander.ml create mode 100644 v2v/var_expander.mli create mode 100644 v2v/var_expander_tests.ml -- 2.20.1

2 5

New patch/feature for libguestfs to parse query string in URL
by Chintan Patel 01 Apr '19

01 Apr '19

Hi Team, We have some change for libguestfs to include query string in URL. The changes are merged with the current Official upstream repo into fork below. https://github.com/chintanrp/libguestfs These changes help to add query string argument in add-drive and use by URI parser. Please let us know if you need more details. And what are the next steps to add these changes to upstream master? Thanks, Chintan

2 2