[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin
This feature allows you to use different image formats for the fixed
appliance. The raw format is used by default.
Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com>
---
lib/launch-direct.c | 2 ++
lib/launch-libvirt.c | 19 ++++++++++++-------
m4/guestfs_appliance.m4 | 11 +++++++++++
3 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/lib/launch-direct.c b/lib/launch-direct.c
index 0be662e25..b9b54857a 100644
--- a/lib/launch-direct.c
+++ b/lib/launch-direct.c
@@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg)
append_list ("id=appliance");
append_list ("cache=unsafe");
append_list ("if=none");
+#ifndef APPLIANCE_FMT_AUTO
append_list ("format=raw");
+#endif
} end_list ();
start_list ("-device") {
append_list ("scsi-hd");
diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c
index 4adb2cfb3..030ea6911 100644
--- a/lib/launch-libvirt.c
+++ b/lib/launch-libvirt.c
@@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv)
/**
* Create a qcow2 format overlay, with the given C<backing_drive>
- * (file). The C<format> parameter, which must be non-NULL, is the
- * backing file format. This is used to create the appliance overlay,
- * and also for read-only drives.
+ * (file). The C<format> parameter is the backing file format.
+ * The C<format> parameter can be NULL, in this case the backing
+ * format will be determined automatically. This is used to create
+ * the appliance overlay, and also for read-only drives.
*/
static char *
make_qcow2_overlay (guestfs_h *g, const char *backing_drive,
@@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive,
char *overlay;
struct guestfs_disk_create_argv optargs;
- assert (format != NULL);
-
if (guestfs_int_lazy_make_tmpdir (g) == -1)
return NULL;
@@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive,
optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK;
optargs.backingfile = backing_drive;
- optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK;
- optargs.backingformat = format;
+ if (format) {
+ optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK;
+ optargs.backingformat = format;
+ }
if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) {
free (overlay);
@@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri)
/* Note that appliance can be NULL if using the old-style appliance. */
if (appliance) {
+#ifdef APPLIANCE_FMT_AUTO
+ params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL);
+#else
params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw");
+#endif
if (!params.appliance_overlay)
goto cleanup;
}
diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4
index 81c43879f..4e1ec8135 100644
--- a/m4/guestfs_appliance.m4
+++ b/m4/guestfs_appliance.m4
@@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH])
AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"],
[Define guestfs default path.])
+
+AC_ARG_ENABLE([appliance-fmt-auto],
+ [AS_HELP_STRING([--enable-appliance-fmt-auto],
+ [enable autodetection of appliance image format @<:@default=no@:>@])],
+ [ENABLE_APPLIANCE_FMT_AUTO="$enableval"],
+ [ENABLE_APPLIANCE_FMT_AUTO=no])
+
+if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then
+ AC_DEFINE([APPLIANCE_FMT_AUTO], [1],
+ [Define to 1 if enabled autodetection of appliance image format.])
+fi
--
2.13.0
4 years, 10 months
1.39 proposal: Let's split up the libguestfs git repo and tarballs
by Richard W.M. Jones
My contention is that the libguestfs git repository is too large and
unwieldy. There are too many separate, unrelated projects and as a
result of that the source has too many dependencies and takes too long
to build and test.
The project divides (sort of) naturally into layers -- the library,
the bindings, the various virt tools -- and could be split along those
lines into separate projects which can then be released and evolve at
their own pace.
My suggested split would be something like this:
* libguestfs: The library, daemon and appliance. That would include
the following directories in a single project:
appliance
bash
contrib
daemon
docs
examples
gnulib
lib
logo
test-tool
tmp
utils
website
* 1 project for each language binding:
csharp
erlang
gobject
golang
haskell
java
lua
ocaml
php
perl
python
ruby
* virt-customize and related tools, we'd probably call this subproject
"virt-builder". It would include virt-builder, virt-customize and
virt-sysprep, since they share a lot of common code.
* 1 project for each of the following items:
small tools written in C
(virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail,
virt-diff, virt-edit, virt-format, guestmount, virt-inspector,
virt-make-fs, virt-rescue)
guestfish
virt-alignment-scan and virt-df
virt-dib
virt-get-kernel
virt-resize
virt-sparsify
virt-v2v and virt-p2v
virt-win-reg
* I'd be inclined to drop the legacy Perl tools virt-tar,
virt-list-filesystems, virt-list-partitions unless someone
especially wished to step forward to maintain them.
* common code and generator: Off to the side we'd somehow need to
package up the common code and the generator for use by all of the
above projects. It wouldn't be a separate project for downstream
packagers, but instead the code would be included (ie. duplicated)
in tarballs and upstream available as a side git repo that you'd
need to include when building (git submodule?). This is somewhat
unspecified.
M4, PO, and tests would be split between the projects as appropriate.
My proposal would be to do this incrementally, rather than all at
once, moving the easier things out first.
Thoughts?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
5 years
[PATCH] v2v: rhv-upload-plugin - improve wait logic after finalize (RHBZ#1680361)
by Daniel Erez
After invoking transfer_service.finalize, check operation
status by examining ImageTransferPhase and DiskStatus.
This is done instead of failing after a predefined timeout
regardless the status.
* not verified *
Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1680361
---
v2v/rhv-upload-plugin.py | 26 +++++++++++++++++++++-----
1 file changed, 21 insertions(+), 5 deletions(-)
diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py
index 2a950c5ed..873c11ce1 100644
--- a/v2v/rhv-upload-plugin.py
+++ b/v2v/rhv-upload-plugin.py
@@ -523,14 +523,30 @@ def close(h):
# waiting for the transfer object to cease to exist, which
# falls through to the exception case and then we can
# continue.
- endt = time.time() + timeout
+ start = time.time()
try:
while True:
time.sleep(1)
- tmp = transfer_service.get()
- if time.time() > endt:
- raise RuntimeError("timed out waiting for transfer "
- "to finalize")
+ transfer = transfer_service.get()
+
+ if transfer is None:
+ disk_service = h['disk_service']
+ disk = disk_service.get()
+ if disk.status == types.DiskStatus.OK:
+ continue
+
+ if transfer.phase == types.ImageTransferPhase.FINISHED_SUCCESS:
+ debug("finalized after %s seconds", time.time() - start)
+ break
+
+ if transfer.phase ==
types.ImageTransferPhase.FINALIZING_SUCCESS:
+ if time.time() > start + timeout:
+ raise RuntimeError("timed out waiting for transfer "
+ "to finalize")
+ continue
+
+ raise RuntimeError("Unexpected transfer phase while
finalizing "
+ "upload %r" % transfer.phase)
except sdk.NotFoundError:
pass
--
5 years, 4 months
[libnbd] Simultaneous read and write
by Richard W.M. Jones
This is a continuation of a discussion we were having on IRC. The
problems with IRC are it's not recorded and it's hard to have deep
technical conversations. I hope this is a decent summary.
Problem simply stated: Certain NBD servers (qemu-nbd in particular)
are able to simultaneously read and write on a socket. ie. They can
be simultaneously reading a request and writing the reply to a
previous request. However libnbd is unable to do this trick.
Although multiple requests can be in flight, libnbd is either writing
on the socket or reading from the socket, but never both.
Visualized it looks like this:
write write write
|===========|______________|=======|_______|===| --> rq to server
_____________|============|_________|=====|_____ <-- rp from server
read read
whereas an ideal libnbd which could write and read simultaneously,
coupled with a server which can do the same:
write
|===========||=======||===| --> requests to server
|============||=====||===== <-- replies from server
read
Eric already observed through testing that the ideal client can be up
to twice as fast as libnbd, as is obvious from the diagram.
The reasons why libnbd can't do this are:
Problem (a) : Only one thread of control can hold the libnbd handle
lock (h->lock), and since there can only be one thread of control
running in each handle at any time, that thread can only be reading or
writing.
Problem (b) : There is only one state machine per handle (h->state),
whereas to handle the write and read sides separately requires two
state machines. In the IRC discussion we gave these the preliminary
names h->wstate and h->rstate.
----------------------------------------------------------------------
It's worth also saying how the current API works, although we might
want to change it.
You grab the underlying file descriptor using nbd_aio_get_fd, which is
what you poll on. You also have to call nbd_aio_get_direction which
returns READ, WRITE or BOTH (== READ|WRITE). You then set up some
events mechanism (eg. poll, epoll, etc.), poll until the file
descriptor is ready, and call one of nbd_aio_notify_read or
nbd_aio_notify_write.
The direction can change any time the handle state changes, which
includes whenever you issue a command (eg. nbd_aio_pread), or whenever
you call nbd_aio_notify_*. You therefore have to call
nbd_aio_get_direction frequently.
A typical loop using poll might look like:
fd = nbd_aio_get_fd (nbd);
for (;;) {
/* <-- If you need to issue more commands, do that here. */
dir = nbd_aio_get_direction (nbd);
pollfd[0].fd = fd;
pollfd[0].events = 0;
if (dir & LIBNBD_AIO_DIRECTION_READ) pollfd[0].events |= POLLIN;
if (dir & LIBNBD_AIO_DIRECTION_WRITE) pollfd[0].events |= POLLOUT;
poll (pollfd, 1, -1);
if (pollfd[0].revents & LIBNBD_AIO_DIRECTION_READ)
nbd_aio_notify_read ();
else if (pollfd[0].revents & LIBNBD_AIO_DIRECTION_WRITE)
nbd_aio_notify_write ();
}
----------------------------------------------------------------------
The above code is of course assuming a single thread. But to
simultaneously read and write we will need at least two threads.
It's hard for me to see a nice way to evolve the API to support
multiple threads, but I guess issues include:
- If one thread is waiting in poll(2) and another thread issues a
command, how do we get the first thread to return from poll and
reevaluate direction? (Eric suggests a pipe-to-self for this)
- If two threads are waiting in poll(2) and both are notified that
the fd is ready, how do we get one of them to read and the other to
write? I think this implies that we don't have two threads doing
poll(2), but if not how do we farm out the read/write work to two
or more threads?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
5 years, 6 months
[PATCH] python: PYTHON_LIBS is not set in Python 3.8 (RHBZ#1705482).
by Richard W.M. Jones
Python 3.8 no longer links C extensions to -lpython, instead relying
on the fact that the python binary itself already contains those
symbols. This means $PYTHON_LIBS is empty and so the Python bindings
are not built.
Use a different test to see if the python module is available.
---
m4/guestfs-python.m4 | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/m4/guestfs-python.m4 b/m4/guestfs-python.m4
index 7d4c991bb..befa9b102 100644
--- a/m4/guestfs-python.m4
+++ b/m4/guestfs-python.m4
@@ -33,14 +33,17 @@ AS_IF([test "x$enable_python" != "xno"],[
PYTHON_VERSION_MINOR=`$PYTHON -c "import sys; print (sys.version_info@<:@1@:>@)"`
PYTHON_VERSION="$PYTHON_VERSION_MAJOR.$PYTHON_VERSION_MINOR"
AC_MSG_RESULT([$PYTHON_VERSION])
+
# Debian: python-2.7.pc, python-3.2.pc
PKG_CHECK_MODULES([PYTHON], [python-"$PYTHON_VERSION"],[
+ have_python_module=1
AC_SUBST([PYTHON_CFLAGS])
AC_SUBST([PYTHON_LIBS])
AC_SUBST([PYTHON_VERSION])
AC_DEFINE([HAVE_PYTHON],[1],[Python library found at compile time])
],[
PKG_CHECK_MODULES([PYTHON], [python],[
+ have_python_module=1
AC_SUBST([PYTHON_CFLAGS])
AC_SUBST([PYTHON_LIBS])
AC_SUBST([PYTHON_VERSION])
@@ -49,6 +52,7 @@ AS_IF([test "x$enable_python" != "xno"],[
AC_MSG_WARN([python $PYTHON_VERSION not found])
])
])
+
AC_MSG_CHECKING([Python prefix])
PYTHON_PREFIX=`$PYTHON -c "import sys; print (sys.prefix)"`
AC_MSG_RESULT([$PYTHON_PREFIX])
@@ -101,4 +105,4 @@ AS_IF([test "x$enable_python" != "xno"],[
AC_SUBST(PYTHON_EXT_SUFFIX)
])
AM_CONDITIONAL([HAVE_PYTHON],
- [test "x$PYTHON" != "xno" && test "x$PYTHON_LIBS" != "x" ])
+ [test "x$PYTHON" != "xno" && test "x$have_python_module" = "x1" ])
--
2.21.0
5 years, 6 months
[nbdkit PATCH 0/4] Play with libnbd for nbdkit-add
by Eric Blake
Patch 1 played with an early draft of Rich's Fedora 30 libnbd package:
https://bugzilla.redhat.com/show_bug.cgi?id=1713767#c17
Note that comment 21 provides a newer package 0.1.1-1 with a different
API; and that libnbd has more unreleased API changes in the pipeline
(whether that will be called 0.2 or 0.1.2); so we'll have to tweak
things based on what is actually available in distros.
Patch 2 is mechanical, to try and make patch 3 a little easier to follow.
Patch 3 shows that libnbd is currently feature-compatible with my
hand-rolled nbd client, albeit with an order-of-magnitude slowdown that
we'll have to investigate.
And patch 4 is my driving reason for this series - it solves a TODO
item about allowing the NBD plugin to be the symmetric solution for
connecting a new server that requires encryption to an old client that
can't speak it. (Although it's late enough at night that I have not
actually yet added the tests to 'make check' to exercise it.)
Also available at https://repo.or.cz/nbdkit/ericb.git under tag
use-libnbd-v1
Eric Blake (4):
nbd: Check for libnbd
nbd: s/nbd_/nbdplug_/
nbd: Use libnbd 0.1
nbd: Add TLS client support
plugins/nbd/nbdkit-nbd-plugin.pod | 77 +-
configure.ac | 20 +
plugins/nbd/nbd-standalone.c | 1364 +++++++++++++++++++++++++++++
plugins/nbd/nbd.c | 1308 +++++++++------------------
TODO | 13 +-
plugins/nbd/Makefile.am | 25 +-
6 files changed, 1860 insertions(+), 947 deletions(-)
create mode 100644 plugins/nbd/nbd-standalone.c
--
2.20.1
5 years, 6 months
[PATCH nbdkit 1/2] nbd: Fix -Werror=maybe-uninitialized warning.
by Richard W.M. Jones
GCC is concerned that if we never go round the loop then fd will be
uninitialized. By asserting that getaddrinfo set result != NULL we
can avoid this.
nbd.c: In function ‘nbd_open_handle’:
nbd.c:974:5: error: ‘fd’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
974 | close (fd);
| ^~~~~~~~~~
nbd.c:954:7: note: ‘fd’ was declared here
954 | int fd;
| ^~
cc1: all warnings being treated as errors
---
plugins/nbd/nbd.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/plugins/nbd/nbd.c b/plugins/nbd/nbd.c
index dba46f1..4d0a67c 100644
--- a/plugins/nbd/nbd.c
+++ b/plugins/nbd/nbd.c
@@ -960,6 +960,8 @@ nbd_connect_tcp (void)
return -1;
}
+ assert (result != NULL);
+
for (rp = result; rp; rp = rp->ai_next) {
fd = socket (rp->ai_family, rp->ai_socktype, rp->ai_protocol);
if (fd == -1)
--
2.21.0
5 years, 6 months
[PATCH 0/5] RFC: switch augeas APIs to OCaml
by Pino Toscano
This synchronizes the embedded ocaml-augeas copy, and reimplements the
augeas APIs using it (dropping all the C code).
The behaviour seems unchanged, although I may have not tested all the
various corner cases.
Pino Toscano (5):
common/mlaugeas: Synchronize with latest ocaml-augeas
daemon: fix/enhance error reporting of Augeas exceptions
Revert "Revert "daemon: implement OptString for OCaml APIs""
daemon: move augeas APIs to OCaml
daemon: drop usage of C augeas library
.gitignore | 1 +
common/mlaugeas/augeas-c.c | 190 ++++++++++++--
common/mlaugeas/augeas.ml | 18 +-
common/mlaugeas/augeas.mli | 32 ++-
daemon/Makefile.am | 7 +-
daemon/aug.ml | 143 +++++++++++
daemon/aug_utils.ml | 42 +++
daemon/aug_utils.mli | 32 +++
daemon/augeas.c | 493 ------------------------------------
daemon/cleanups.c | 11 -
daemon/daemon-c.c | 29 ++-
daemon/daemon-c.h | 1 +
daemon/daemon.h | 35 ---
daemon/lvm-filter.c | 2 -
daemon/mount.c | 3 +-
daemon/mount_utils.ml | 2 +-
docs/C_SOURCE_FILES | 1 -
generator/actions_augeas.ml | 15 ++
generator/actions_core.ml | 2 +
generator/daemon.ml | 18 +-
po/POTFILES | 1 -
21 files changed, 498 insertions(+), 580 deletions(-)
create mode 100644 daemon/aug.ml
create mode 100644 daemon/aug_utils.ml
create mode 100644 daemon/aug_utils.mli
delete mode 100644 daemon/augeas.c
--
2.21.0
5 years, 6 months
[PATCH] appliance: Remove /etc/dhcp/dhclient-enter-hooks.d/resolved.
by Richard W.M. Jones
Workaround for Ubuntu which uses this script to try to start a systemd
service. That won't work because systemd is not used inside the
appliance. See:
https://bugs.launchpad.net/ubuntu/+source/supermin/+bug/1824236
Thanks: Ioanna Alifieraki
---
appliance/init | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/appliance/init b/appliance/init
index 4f2b55822..b1c4d09ea 100755
--- a/appliance/init
+++ b/appliance/init
@@ -122,7 +122,9 @@ ip link set dev lo up
if test "$guestfs_network" = 1; then
iface=$(ls -I all -I default -I lo /proc/sys/net/ipv4/conf)
- touch /etc/fstab # Workaround for Ubuntu.
+ # Two workarounds for Ubuntu:
+ touch /etc/fstab
+ rm -f /etc/dhcp/dhclient-enter-hooks.d/resolved
if dhclient --version >/dev/null 2>&1; then
dhclient $iface
else
--
2.20.1
5 years, 7 months