[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin
This feature allows you to use different image formats for the fixed
appliance. The raw format is used by default.
Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com>
---
lib/launch-direct.c | 2 ++
lib/launch-libvirt.c | 19 ++++++++++++-------
m4/guestfs_appliance.m4 | 11 +++++++++++
3 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/lib/launch-direct.c b/lib/launch-direct.c
index 0be662e25..b9b54857a 100644
--- a/lib/launch-direct.c
+++ b/lib/launch-direct.c
@@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg)
append_list ("id=appliance");
append_list ("cache=unsafe");
append_list ("if=none");
+#ifndef APPLIANCE_FMT_AUTO
append_list ("format=raw");
+#endif
} end_list ();
start_list ("-device") {
append_list ("scsi-hd");
diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c
index 4adb2cfb3..030ea6911 100644
--- a/lib/launch-libvirt.c
+++ b/lib/launch-libvirt.c
@@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv)
/**
* Create a qcow2 format overlay, with the given C<backing_drive>
- * (file). The C<format> parameter, which must be non-NULL, is the
- * backing file format. This is used to create the appliance overlay,
- * and also for read-only drives.
+ * (file). The C<format> parameter is the backing file format.
+ * The C<format> parameter can be NULL, in this case the backing
+ * format will be determined automatically. This is used to create
+ * the appliance overlay, and also for read-only drives.
*/
static char *
make_qcow2_overlay (guestfs_h *g, const char *backing_drive,
@@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive,
char *overlay;
struct guestfs_disk_create_argv optargs;
- assert (format != NULL);
-
if (guestfs_int_lazy_make_tmpdir (g) == -1)
return NULL;
@@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive,
optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK;
optargs.backingfile = backing_drive;
- optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK;
- optargs.backingformat = format;
+ if (format) {
+ optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK;
+ optargs.backingformat = format;
+ }
if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) {
free (overlay);
@@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri)
/* Note that appliance can be NULL if using the old-style appliance. */
if (appliance) {
+#ifdef APPLIANCE_FMT_AUTO
+ params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL);
+#else
params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw");
+#endif
if (!params.appliance_overlay)
goto cleanup;
}
diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4
index 81c43879f..4e1ec8135 100644
--- a/m4/guestfs_appliance.m4
+++ b/m4/guestfs_appliance.m4
@@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH])
AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"],
[Define guestfs default path.])
+
+AC_ARG_ENABLE([appliance-fmt-auto],
+ [AS_HELP_STRING([--enable-appliance-fmt-auto],
+ [enable autodetection of appliance image format @<:@default=no@:>@])],
+ [ENABLE_APPLIANCE_FMT_AUTO="$enableval"],
+ [ENABLE_APPLIANCE_FMT_AUTO=no])
+
+if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then
+ AC_DEFINE([APPLIANCE_FMT_AUTO], [1],
+ [Define to 1 if enabled autodetection of appliance image format.])
+fi
--
2.13.0
4 years, 9 months
1.39 proposal: Let's split up the libguestfs git repo and tarballs
by Richard W.M. Jones
My contention is that the libguestfs git repository is too large and
unwieldy. There are too many separate, unrelated projects and as a
result of that the source has too many dependencies and takes too long
to build and test.
The project divides (sort of) naturally into layers -- the library,
the bindings, the various virt tools -- and could be split along those
lines into separate projects which can then be released and evolve at
their own pace.
My suggested split would be something like this:
* libguestfs: The library, daemon and appliance. That would include
the following directories in a single project:
appliance
bash
contrib
daemon
docs
examples
gnulib
lib
logo
test-tool
tmp
utils
website
* 1 project for each language binding:
csharp
erlang
gobject
golang
haskell
java
lua
ocaml
php
perl
python
ruby
* virt-customize and related tools, we'd probably call this subproject
"virt-builder". It would include virt-builder, virt-customize and
virt-sysprep, since they share a lot of common code.
* 1 project for each of the following items:
small tools written in C
(virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail,
virt-diff, virt-edit, virt-format, guestmount, virt-inspector,
virt-make-fs, virt-rescue)
guestfish
virt-alignment-scan and virt-df
virt-dib
virt-get-kernel
virt-resize
virt-sparsify
virt-v2v and virt-p2v
virt-win-reg
* I'd be inclined to drop the legacy Perl tools virt-tar,
virt-list-filesystems, virt-list-partitions unless someone
especially wished to step forward to maintain them.
* common code and generator: Off to the side we'd somehow need to
package up the common code and the generator for use by all of the
above projects. It wouldn't be a separate project for downstream
packagers, but instead the code would be included (ie. duplicated)
in tarballs and upstream available as a side git repo that you'd
need to include when building (git submodule?). This is somewhat
unspecified.
M4, PO, and tests would be split between the projects as appropriate.
My proposal would be to do this incrementally, rather than all at
once, moving the easier things out first.
Thoughts?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
4 years, 11 months
[PATCH] v2v: rhv-upload-plugin - improve wait logic after finalize (RHBZ#1680361)
by Daniel Erez
After invoking transfer_service.finalize, check operation
status by examining ImageTransferPhase and DiskStatus.
This is done instead of failing after a predefined timeout
regardless the status.
* not verified *
Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1680361
---
v2v/rhv-upload-plugin.py | 26 +++++++++++++++++++++-----
1 file changed, 21 insertions(+), 5 deletions(-)
diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py
index 2a950c5ed..873c11ce1 100644
--- a/v2v/rhv-upload-plugin.py
+++ b/v2v/rhv-upload-plugin.py
@@ -523,14 +523,30 @@ def close(h):
# waiting for the transfer object to cease to exist, which
# falls through to the exception case and then we can
# continue.
- endt = time.time() + timeout
+ start = time.time()
try:
while True:
time.sleep(1)
- tmp = transfer_service.get()
- if time.time() > endt:
- raise RuntimeError("timed out waiting for transfer "
- "to finalize")
+ transfer = transfer_service.get()
+
+ if transfer is None:
+ disk_service = h['disk_service']
+ disk = disk_service.get()
+ if disk.status == types.DiskStatus.OK:
+ continue
+
+ if transfer.phase == types.ImageTransferPhase.FINISHED_SUCCESS:
+ debug("finalized after %s seconds", time.time() - start)
+ break
+
+ if transfer.phase ==
types.ImageTransferPhase.FINALIZING_SUCCESS:
+ if time.time() > start + timeout:
+ raise RuntimeError("timed out waiting for transfer "
+ "to finalize")
+ continue
+
+ raise RuntimeError("Unexpected transfer phase while
finalizing "
+ "upload %r" % transfer.phase)
except sdk.NotFoundError:
pass
--
5 years, 3 months
[PATCH] appliance: Remove /etc/dhcp/dhclient-enter-hooks.d/resolved.
by Richard W.M. Jones
Workaround for Ubuntu which uses this script to try to start a systemd
service. That won't work because systemd is not used inside the
appliance. See:
https://bugs.launchpad.net/ubuntu/+source/supermin/+bug/1824236
Thanks: Ioanna Alifieraki
---
appliance/init | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/appliance/init b/appliance/init
index 4f2b55822..b1c4d09ea 100755
--- a/appliance/init
+++ b/appliance/init
@@ -122,7 +122,9 @@ ip link set dev lo up
if test "$guestfs_network" = 1; then
iface=$(ls -I all -I default -I lo /proc/sys/net/ipv4/conf)
- touch /etc/fstab # Workaround for Ubuntu.
+ # Two workarounds for Ubuntu:
+ touch /etc/fstab
+ rm -f /etc/dhcp/dhclient-enter-hooks.d/resolved
if dhclient --version >/dev/null 2>&1; then
dhclient $iface
else
--
2.20.1
5 years, 5 months
[PATCH nbdkit 0/7] server: Implement NBD_FLAG_CAN_MULTI_CONN.
by Richard W.M. Jones
First thing to say is that I need to do a *lot* more testing on this,
so this is just an early peek. In particular, although it passed
‘make check && make check-valgrind’ I have *not* tested it against a
multi-conn-aware client such as the Linux kernel >= 4.9.
This implements NBD_FLAG_CAN_MULTI_CONN, described in the protocol doc
as:
"NBD_FLAG_CAN_MULTI_CONN: Indicates that the server operates
entirely without cache, or that the cache it uses is shared among
all connections to the given device. In particular, if this flag is
present, then the effects of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA MUST
be visible across all connections when the server sends its reply to
that command to the client. In the absence of this flag, clients
SHOULD NOT multiplex their commands over more than one connection to
the export."
This is necessary to support the Linux nbd client -C option.
The only plugin which sets the flag so far is file. The ocaml, sh and
nbd plugins allow the flag to be controlled or passed through.
Notable also is that the blocksize filter has to filter out this flag,
because I'm not convinced that the bounce buffer is safe. However I
believe the other filters *are* safe (although not really certain
about the fua filter, and the new cache filter is tricky too).
My feeling is that we should set the flag unconditionally for all
readonly connections, but I can think of nasty corner cases where it
might not work. We should most probably set it in all plugins that
are readonly (eg. nbdkit-pattern-plugin).
Rich.
5 years, 6 months
[nbdkit PATCH 0/7] Implement structured replies in nbd plugin
by Eric Blake
I'm hoping to implement .extents for the nbd plugin; this is a
prerequisite. I'm not sure about patch 3 - if we like it, I'll
squash it to 2, if we don't, I think we are okay just dropping
it. I'm also wondering if we have to worry about malicious
plugins that don't populate the entire .pread buffer in an
effort to get nbdkit to expose portions of the heap; my patch 7
loses some time on extra memset()s to avoid that, although it's
probably in the noise compared to the time spent talking over
sockets to the remote server.
Eric Blake (7):
protocol: Send correct structured reply error for non-Linux
protocol: Support EOVERFLOW
RFC: protocol: Only send EOVERFLOW when valid
nbd: Honor server global flags on little-endian
protocol: Add helpers for error response handling
nbd: Implement NBD_OPT_GO client request
nbd: Implement structured replies
common/protocol/protocol.h | 21 +-
plugins/nbd/nbd.c | 434 +++++++++++++++++++++++++++++++++----
server/protocol.c | 16 +-
3 files changed, 422 insertions(+), 49 deletions(-)
--
2.20.1
5 years, 6 months
[PATCH 43 0/7] v2v: switch to ocaml-libvirt
by Pino Toscano
Hi,
this series switches virt-2v to ocaml-libvirt, embedding the latest
version of it from git. This way, it is possible to improve the way
v2v connects to libvirt for both input, and output modules, and
interacts with libvirt (e.g. no more virsh calls needed in virt-v2v).
As side effect, virt-v2v now requires libvirt, as keeping it optional
would create too much burden.
I could not test all the libvirt input modes (like VDDK, and Xen), but
VMware and libvirtxml work fine as before.
Changes from v3:
- rebase on master
- update libvirt-ocaml from its master branch
- use $(srcdir) in common/mllibvirt/Makefile.am
- parse_libvirt_xml: open a libvirt connection only when needed (and
only once)
Changes from v2:
- rebase on master
Changes from v1:
- rebase on master
- update ocaml-libvirt from libvirt-ocaml.git on libvirt.org, and adjust
the code to it
- pass again the URI to input_libvirt_vddk, so an error message is
preserved
Pino Toscano (7):
v2v: require libvirt
common: Bundle the libvirt-ocaml library for use by virt-v2v
v2v: switch to ocaml-libvirt
v2v: -o libvirt: use a Lazy for the connection
v2v: -o libvirt: switch away from virsh
v2v: test-harness: stop using the external ocaml-libvirt
build: stop looking for ocaml-libvirt
.gitignore | 2 +
Makefile.am | 5 +-
common/mllibvirt/Makefile.am | 102 ++
common/mllibvirt/generator.pl | 908 +++++++++++++
common/mllibvirt/libvirt.README | 9 +
common/mllibvirt/libvirt.ml | 1673 ++++++++++++++++++++++++
common/mllibvirt/libvirt.mli | 1647 ++++++++++++++++++++++++
common/mllibvirt/libvirt_c_epilogue.c | 462 +++++++
common/mllibvirt/libvirt_c_oneoffs.c | 1714 +++++++++++++++++++++++++
common/mllibvirt/libvirt_c_prologue.c | 134 ++
configure.ac | 1 +
docs/C_SOURCE_FILES | 1 -
m4/guestfs-ocaml.m4 | 4 -
po/POTFILES | 1 -
v2v/Makefile.am | 26 +-
v2v/copy_to_local.ml | 7 +-
v2v/input_libvirt.ml | 20 +-
v2v/input_libvirt_other.ml | 27 +-
v2v/input_libvirt_other.mli | 5 +-
v2v/input_libvirt_vcenter_https.ml | 13 +-
v2v/input_libvirt_vcenter_https.mli | 2 +-
v2v/input_libvirt_vddk.ml | 15 +-
v2v/input_libvirt_vddk.mli | 4 +-
v2v/input_libvirt_xen_ssh.ml | 13 +-
v2v/input_libvirt_xen_ssh.mli | 2 +-
v2v/libvirt_utils-c.c | 539 --------
v2v/libvirt_utils.ml | 95 +-
v2v/libvirt_utils.mli | 51 +-
v2v/output_libvirt.ml | 56 +-
v2v/parse_libvirt_xml.ml | 23 +-
v2v/parse_libvirt_xml.mli | 11 +-
v2v/test-harness/Makefile.am | 5 +-
v2v/v2v.ml | 12 +-
33 files changed, 6893 insertions(+), 696 deletions(-)
create mode 100644 common/mllibvirt/Makefile.am
create mode 100755 common/mllibvirt/generator.pl
create mode 100644 common/mllibvirt/libvirt.README
create mode 100644 common/mllibvirt/libvirt.ml
create mode 100644 common/mllibvirt/libvirt.mli
create mode 100644 common/mllibvirt/libvirt_c_epilogue.c
create mode 100644 common/mllibvirt/libvirt_c_oneoffs.c
create mode 100644 common/mllibvirt/libvirt_c_prologue.c
delete mode 100644 v2v/libvirt_utils-c.c
--
2.20.1
5 years, 6 months
[PATCH 00/11] v2v: Change virt-v2v to use nbdkit for input in several modes.
by Richard W.M. Jones
This series (except the last one) changes virt-v2v to use nbdkit for
several input modes:
-i vmx -it vddk: No change in functionality, as this already uses
nbdkit-vddk-plugin, but the code is refactored for the other modes to
use.
-i libvirtxml: Use nbdkit-curl-plugin instead of qemu curl.
vCenter: Use nbdkit-curl-plugin instead of qemu curl.
xen: Use nbdkit-ssh-plugin instead of qemu ssh.
-i vmx -it ssh: Use nbdkit-ssh-plugin instead of qemu ssh.
I was hoping to post this patch series when it was longer and showed
some benefits. However those patches aren't ready quite yet, but
these ones are ready for review. However some of the future benefits
will be:
* using nbdkit-rate-plugin to bandwidth limit the input side
* better support for modern SSH encryption
* possibility of SSH password auth (instead of requiring ssh-agent)
I have tested this locally. Unfortunately none of the existing tests
cover any of these input cases.
The last patch reduces the minimum required version of nbdkit from
1.12 (not yet available) to 1.11, so this patch can be tested in
Fedora 30. It is obviously not meant to be applied.
Rich.
5 years, 6 months
[nbdkit PATCH 0/3] Fix data integrity in vddk plugin
by Martin Kletzander
Couple of fixes to return correct data and one nice-to-have clean-up which is not needed. I just find it nicer to read.
Martin Kletzander (3):
vddk: Use a separate handle for single-link=true
vddk: Do not report hole extents to be zero with single-link=true
vddk: Eliminate one needless goto
plugins/vddk/vddk.c | 48 +++++++++++++++++++++++++++++++++------------
1 file changed, 36 insertions(+), 12 deletions(-)
--
2.21.0
5 years, 6 months
[nbdkit PATCH 0/2] Let nbd plugin connect to TCP socket
by Eric Blake
Accepting only Unix sockets can be a bit limiting; let's be more flexible.
Eric Blake (2):
nbd: Refactor Unix socket connection
nbd: Support TCP socket
plugins/nbd/nbdkit-nbd-plugin.pod | 36 ++++--
plugins/nbd/nbd.c | 175 ++++++++++++++++++++++--------
TODO | 3 -
3 files changed, 161 insertions(+), 53 deletions(-)
--
2.20.1
5 years, 6 months
