June 2021 - Libguestfs - libguestfs List Archives

nbdcpy: from scratch nbdcopy using io_uring
by Abhay Raj Singh 25 Aug '21

25 Aug '21

In a previous email, Mr. Jones suggested, writing a test implementation of nbdcopy from scratch. First of all, thank you, Mr. Jones, as I got a better perspective of the problem and the solution. I have almost written the bare-bones implementation for handling NBD-related stuff on which I will base the io_uring core piece by piece. It is written in C++, as I am more comfortable with it. But, made it easily translatable to C. I have been working on a solution and have documented the current approach, it would be great if you have a look at it and let me know what you think. Approach document: https://gitlab.com/rathod-sahaab/nbdcpy/-/blob/dev/docs/SOLUTION.md Thanks and regards, Abhay

3 16

[PATCH] point users to Libera Chat rather than FreeNode
by Daniel P. Berrangé 29 Jul '21

29 Jul '21

Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- docs/guestfs-faq.pod | 2 +- website/index.html.in | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/guestfs-faq.pod b/docs/guestfs-faq.pod index bea609591..15b1247b0 100644 --- a/docs/guestfs-faq.pod +++ b/docs/guestfs-faq.pod @@ -108,7 +108,7 @@ There is a mailing list, mainly for development, but users are also welcome to ask questions about libguestfs and the virt tools: L<https://www.redhat.com/mailman/listinfo/libguestfs> -You can also talk to us on IRC channel C<#libguestfs> on FreeNode. +You can also talk to us on IRC channel C<#guestfs> on Libera Chat. We're not always around, so please stay in the channel after asking your question and someone will get back to you. diff --git a/website/index.html.in b/website/index.html.in index f469c5eeb..7453129d6 100644 --- a/website/index.html.in +++ b/website/index.html.in @@ -55,8 +55,8 @@ guestfish --ro -i -a disk.img <p> Join us on the <a href="http://www.redhat.com/mailman/listinfo/libguestfs">libguestfs -mailing list</a>, or on IRC channel <code>#libguestfs</code> -on <a href="http://freenode.net/">FreeNode</a>. +mailing list</a>, or on IRC channel <code>#guestfs</code> +on <a href="https://libera.chat/">Libera Chat</a>. </p> </div> -- 2.31.1

3 4

Figuring out some failing tests for libnbd
by Martin Kletzander 07 Jul '21

07 Jul '21

I am preparing more patches for CI to run check-valgrind and fix ongoing errors but there are two issues I can not identify the reason why they are failing. - On debian-10 the info/info-can.sh started failing and the error message is just one of those I saw earlier in other places: libnbd: debug: nbd1: nbd_opt_abort: leave: error="nbd_opt_abort: invalid state: READY: the handle must be negotiating: Invalid argument" - On Fedora rawhide I hit a random issue where a port in a URI was translated to its name and looking at the code I can not find how this could have happened. Until this is fixed the test suite is unreliable and notification fatigue will cause everyone to start ignoring any future failures. /builds/nertpinx/libnbd/tests/.libs/aio-connect: actual URI nbd://127.0.0.1:altova-lm/ != expected URI nbd://127.0.0.1:35355/ - Both openSUSE builds are failing to run check-valgrind and it looks like it might be unrelated to libnbd, although it would be nice for someone else to confirm that. For now I have disabled check-valgrind on those platforms in my branch. - Similarly to openSUSE Ubuntu 20.04 fails in valgrind tests, but somewhere down the GnuTLS rabbit hole, which I presume is unrelated too, so I disabled check-valgrind on that one as well. I will send the patches once they are cleaned up, but I wanted to let everyone know what the current status is because eliminating all random issues is essential to properly consuming CI results. Thanks, Martin

2 7

[PATCH libnbd] lib/uri.c: nbd_get_uri: Do not translate port name into service
by Richard W.M. Jones 30 Jun '21

30 Jun '21

Because I forgot to use the NI_NUMERICSERV flag, getnameinfo would translate the port number into a service name (when possible) using /etc/services. This is not really desirable since raw port numbers are more portable and descriptive. This also caused occasional test failures when pick-a-port happened to pick a port number which coincided with a service name. This commit also adds a test. Previous to the fix, the test failed with: $ ./run tests/aio-connect-port tests/.libs/aio-connect-port: actual URI nbd://127.0.0.1:sieve-filter/ != expected URI nbd://127.0.0.1:2000/ This test cannot be run as part of the test suite because it requires a fixed port number. See also https://listman.redhat.com/archives/libguestfs/2021-June/msg00205.html Reported-by: Martin Kletzander --- .gitignore | 1 + lib/uri.c | 3 +- tests/Makefile.am | 9 +++ tests/aio-connect-port.c | 128 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 140 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 865b689..602438d 100644 --- a/.gitignore +++ b/.gitignore @@ -155,6 +155,7 @@ Makefile.in /stamp-h1 /test-driver /tests/aio-connect +/tests/aio-connect-port /tests/aio-parallel /tests/aio-parallel-load /tests/aio-parallel-load-tls diff --git a/lib/uri.c b/lib/uri.c index 9b97e46..bcecbad 100644 --- a/lib/uri.c +++ b/lib/uri.c @@ -422,7 +422,8 @@ nbd_unlocked_get_uri (struct nbd_handle *h) uri.scheme = using_tls ? "nbds" : "nbd"; err = getnameinfo ((struct sockaddr *) &h->connaddr, h->connaddrlen, - host, sizeof host, serv, sizeof serv, NI_NUMERICHOST); + host, sizeof host, serv, sizeof serv, + NI_NUMERICHOST | NI_NUMERICSERV); if (err != 0) { set_error (0, "getnameinfo: %s", gai_strerror (err)); goto out; diff --git a/tests/Makefile.am b/tests/Makefile.am index 09de079..aad5513 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -203,6 +203,7 @@ check_PROGRAMS += \ connect-tcp \ connect-tcp6 \ aio-connect \ + aio-connect-port \ aio-parallel \ aio-parallel-load \ synch-parallel \ @@ -252,6 +253,9 @@ TESTS += \ closure-lifetimes \ $(NULL) +# This test is compiled but not run because it requires a fixed port: +# aio-connect-port + errors_SOURCES = errors.c errors_CPPFLAGS = -I$(top_srcdir)/include errors_CFLAGS = $(WARNINGS_CFLAGS) @@ -489,6 +493,11 @@ aio_connect_CPPFLAGS = -I$(top_srcdir)/include aio_connect_CFLAGS = $(WARNINGS_CFLAGS) aio_connect_LDADD = $(top_builddir)/lib/libnbd.la +aio_connect_port_SOURCES = aio-connect-port.c +aio_connect_port_CPPFLAGS = -I$(top_srcdir)/include +aio_connect_port_CFLAGS = $(WARNINGS_CFLAGS) +aio_connect_port_LDADD = $(top_builddir)/lib/libnbd.la + aio_parallel_SOURCES = aio-parallel.c aio_parallel_CPPFLAGS = \ -I$(top_srcdir)/include \ diff --git a/tests/aio-connect-port.c b/tests/aio-connect-port.c new file mode 100644 index 0000000..5f9613e --- /dev/null +++ b/tests/aio-connect-port.c @@ -0,0 +1,128 @@ +/* NBD client library in userspace + * Copyright (C) 2013-2021 Red Hat Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/* Test connecting to a fixed IPv4 TCP port (2000) using + * nbd_aio_connect. This test is compiled but not normally run + * because of the requirement for the fixed port number. + * + * See also: + * https://listman.redhat.com/archives/libguestfs/2021-June/msg00205.html + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <fcntl.h> +#include <unistd.h> +#include <sys/socket.h> +#include <netinet/in.h> +#include <arpa/inet.h> + +#include <libnbd.h> + +#define PIDFILE "aio-connect-port.pid" + +int +main (int argc, char *argv[]) +{ + struct nbd_handle *nbd; + int port = 2000; + const char *port_str = "2000"; + pid_t pid; + size_t i; + struct sockaddr_in addr; + char *actual_uri, *expected_uri; + + unlink (PIDFILE); + + pid = fork (); + if (pid == -1) { + perror ("fork"); + exit (EXIT_FAILURE); + } + if (pid == 0) { + execlp ("nbdkit", + "nbdkit", "-f", "-p", port_str, "-P", PIDFILE, + "--exit-with-parent", "null", NULL); + perror ("nbdkit"); + _exit (EXIT_FAILURE); + } + + /* Wait for nbdkit to start listening. */ + for (i = 0; i < 60; ++i) { + if (access (PIDFILE, F_OK) == 0) + break; + sleep (1); + } + unlink (PIDFILE); + + nbd = nbd_create (); + if (nbd == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + if (nbd_supports_uri (nbd) != 1) { + fprintf (stderr, "skip: compiled without URI support\n"); + exit (77); + } + + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl (INADDR_LOOPBACK); + addr.sin_port = htons (port); + + if (nbd_aio_connect (nbd, (struct sockaddr *) &addr, sizeof addr) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + /* Wait until we have connected. */ + while (!nbd_aio_is_ready (nbd)) { + if (nbd_poll (nbd, -1) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + } + + /* libnbd should be able to construct a URI for this connection. */ + if (asprintf (&expected_uri, "nbd://127.0.0.1:%s/", port_str) == -1) { + perror ("asprintf"); + exit (EXIT_FAILURE); + } + actual_uri = nbd_get_uri (nbd); + if (actual_uri == NULL) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + if (strcmp (actual_uri, expected_uri) != 0) { + fprintf (stderr, "%s: actual URI %s != expected URI %s\n", + argv[0], actual_uri, expected_uri); + exit (EXIT_FAILURE); + } + free (actual_uri); + free (expected_uri); + + if (nbd_shutdown (nbd, 0) == -1) { + fprintf (stderr, "%s\n", nbd_get_error ()); + exit (EXIT_FAILURE); + } + + nbd_close (nbd); + exit (EXIT_SUCCESS); +} -- 2.32.0

2 1

[PATCH libnbd] python: Implement nbd.aio_connect for AF_UNIX
by Richard W.M. Jones 30 Jun '21

30 Jun '21

This call previously just called abort(). Implement it for Unix domain sockets (the easy case). Implementing it for AF_INET and AF_INET6 is more complicated so that is left as a to-do. Note also that implementing this fully for Python is a bit pointless. It would be easier for a Python program to call nbd.aio_connect_tcp(host, port) instead of calling nbd.aio_connect((host, port)). Both cases would do hostname lookups but the former is already implemented. --- generator/Python.ml | 17 +++++---- python/Makefile.am | 6 ++- python/python-aio-connect-unix.sh | 36 ++++++++++++++++++ python/utils.c | 63 +++++++++++++++++++++++++++++++ 4 files changed, 114 insertions(+), 8 deletions(-) diff --git a/generator/Python.ml b/generator/Python.ml index 25f92d7..bcfd4bf 100644 --- a/generator/Python.ml +++ b/generator/Python.ml @@ -41,6 +41,8 @@ struct py_aio_buffer { extern char **nbd_internal_py_get_string_list (PyObject *); extern void nbd_internal_py_free_string_list (char **); +extern int nbd_internal_py_get_sockaddr (PyObject *, + struct sockaddr_storage *, socklen_t *); extern struct py_aio_buffer *nbd_internal_py_get_aio_buffer (PyObject *); static inline struct nbd_handle * @@ -300,10 +302,9 @@ let print_python_binding name { args; optargs; ret; may_set_error } = | SizeT n -> pr " Py_ssize_t %s;\n" n | SockAddrAndLen (n, _) -> - pr " /* XXX Complicated - Python uses a tuple of different\n"; - pr " * lengths for the different socket types.\n"; - pr " */\n"; - pr " PyObject *%s;\n" n + pr " PyObject *%s;\n" n; + pr " struct sockaddr_storage %s_sa;\n" n; + pr " socklen_t %s_len;\n" n; | String n -> pr " const char *%s;\n" n | StringList n -> pr " PyObject *py_%s;\n" n; @@ -440,8 +441,10 @@ let print_python_binding name { args; optargs; ret; may_set_error } = pr " %s = PyBytes_AS_STRING (py_%s);\n" n n; pr " assert (%s != NULL);\n" n | SizeT n -> () - | SockAddrAndLen _ -> - pr " abort (); /* XXX SockAddrAndLen not implemented */\n"; + | SockAddrAndLen (n, _) -> + pr " if (nbd_internal_py_get_sockaddr (%s, &%s_sa, &%s_len) == -1)\n" + n n n; + pr " goto out;\n" | String _ -> () | StringList n -> pr " %s = nbd_internal_py_get_string_list (py_%s);\n" n n; @@ -468,7 +471,7 @@ let print_python_binding name { args; optargs; ret; may_set_error } = | Int64 n -> pr ", %s_i64" n | Path n -> pr ", %s" n | SizeT n -> pr ", (size_t)%s" n - | SockAddrAndLen (n, _) -> pr ", /* XXX */ (void *) %s, 0" n + | SockAddrAndLen (n, _) -> pr ", (struct sockaddr *) &%s_sa, %s_len" n n | String n -> pr ", %s" n | StringList n -> pr ", %s" n | UInt n | UIntPtr n -> pr ", %s" n diff --git a/python/Makefile.am b/python/Makefile.am index 718f891..3779410 100644 --- a/python/Makefile.am +++ b/python/Makefile.am @@ -81,9 +81,13 @@ TESTS_ENVIRONMENT = \ LIBNBD_DEBUG=1 \ MALLOC_CHECK_=1 \ MALLOC_PERTURB_=$(shell bash -c 'echo $$(( 1 + (RANDOM & 255) ))') \ + PYTHON="$(PYTHON)" \ $(NULL) LOG_COMPILER = $(top_builddir)/run -TESTS += run-python-tests +TESTS += \ + python-aio-connect-unix.sh \ + run-python-tests \ + $(NULL) endif HAVE_NBDKIT diff --git a/python/python-aio-connect-unix.sh b/python/python-aio-connect-unix.sh new file mode 100755 index 0000000..7be0f87 --- /dev/null +++ b/python/python-aio-connect-unix.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# nbd client library in userspace +# Copyright (C) 2020-2021 Red Hat Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +. ../tests/functions.sh + +set -e +set -x + +requires nbdkit --version +requires $PYTHON --version + +nbdkit -U - null --run '$PYTHON -c " +import nbd +import sys +h = nbd.NBD() +addr = sys.argv[1] +h.aio_connect(addr) +while h.aio_is_connecting(): + h.poll(1) +assert h.aio_is_ready() +" "$unixsocket"' diff --git a/python/utils.c b/python/utils.c index 0e3164c..37f0c55 100644 --- a/python/utils.c +++ b/python/utils.c @@ -26,6 +26,8 @@ #include <stdio.h> #include <stdlib.h> #include <assert.h> +#include <sys/socket.h> +#include <sys/un.h> #include <libnbd.h> @@ -92,3 +94,64 @@ nbd_internal_py_free_string_list (char **argv) free (argv[i]); free (argv); } + +/* Convert a Python object into a struct sockaddr, according to the + * general rules described here: + * https://docs.python.org/3/library/socket.html + * + * There is a function in cpython called getsockaddrarg which roughly + * does the same thing, but in cpython they know the socket family + * already (which we do not). In any case that function cannot be + * called directly. + */ +int +nbd_internal_py_get_sockaddr (PyObject *addr, + struct sockaddr_storage *ss, socklen_t *len) +{ + memset (ss, 0, sizeof *ss); + + if (PyUnicode_Check (addr)) { /* AF_UNIX */ + struct sockaddr_un *sun = (struct sockaddr_un *)ss; + const char *unixsocket; + size_t namelen; + + sun->sun_family = AF_UNIX; + unixsocket = PyUnicode_AsUTF8 (addr); + if (!unixsocket) + goto err; + namelen = strlen (unixsocket); + if (namelen > sizeof sun->sun_path) { + PyErr_SetString (PyExc_RuntimeError, + "get_sockaddr: Unix domain socket name too long"); + return -1; + } + memcpy (sun->sun_path, unixsocket, namelen); + *len = sizeof *sun; + return 0; + } + +#if 0 + else if (PyTuple_Check (addr)) { + Py_ssize_t n = PyTuple_Size (addr); + + switch (n) { + case 2: /* AF_INET */ + /* XXX TODO */ + break; + + case 4: /* AF_INET6 */ + /* XXX TODO */ + break; + + default: + goto err; + } + } +#endif + + else { + err: + PyErr_SetString (PyExc_TypeError, "get_sockaddr: unknown address type"); + return -1; + } +} -- 2.32.0

2 1

[libnbd PATCH 0/3] CI updates
by Martin Kletzander 30 Jun '21

30 Jun '21

Some fixes plus check-valgrind enablement. Martin Kletzander (3): qemu-storage-daemon 5.2.0 is still broken fuse: move check-valgrind out from condition ci: Rework the build script to run check-valgrind properly interop/interop-qemu-storage-daemon.sh | 3 +- .gitlab-ci.yml | 3 + ci/build.sh | 96 +++++++++++--------- ci/cirrus/freebsd-12.vars | 4 +- ci/cirrus/freebsd-13.vars | 4 +- ci/cirrus/freebsd-current.vars | 4 +- ci/cirrus/macos-11.vars | 5 +- ci/containers/centos-8.Dockerfile | 5 +- ci/containers/centos-stream-8.Dockerfile | 5 +- ci/containers/debian-10.Dockerfile | 5 +- ci/containers/debian-sid.Dockerfile | 5 +- ci/containers/fedora-33.Dockerfile | 7 +- ci/containers/fedora-34.Dockerfile | 7 +- ci/containers/fedora-rawhide.Dockerfile | 7 +- ci/containers/opensuse-leap-152.Dockerfile | 5 +- ci/containers/opensuse-tumbleweed.Dockerfile | 5 +- ci/containers/ubuntu-1804.Dockerfile | 5 +- ci/containers/ubuntu-2004.Dockerfile | 5 +- fuse/Makefile.am | 4 +- 19 files changed, 102 insertions(+), 82 deletions(-) -- 2.32.0

2 4

Virt-v2v split performance
by Richard W.M. Jones 29 Jun '21

29 Jun '21

[Some random mumblings that I want to put on record ...] The new "modular virt-v2v" (which I'll probably release as "virt-v2v 2.0" even though it's not really a rewrite) is in the state where it can convert guests: Alpha-quality code: https://github.com/rwmjones/virt-v2v/commits/2021-virt-v2v-split Original plan: https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html One thing of note is that it's slower in some places and maybe will be faster in others. Because I'm using NBD disk pipelines for everything, when we want to read or write to a local disk image we go through nbdkit (instead of just accessing the local file as virt-v2v would do). However the local file case is not very common in real world usage (except for testing). On the flip side, nbdcopy is faster than qemu-img convert when copying between NBD source and target. In fact when copying from file to file through 2 x nbdkits, nbdcopy is about the same speed as qemu-img convert copying between local files. Some actual timings. As a baseline here's original virt-v2v doing a file to file conversion: $ virt-v2v -i disk /var/tmp/rhel-8.4.img -o disk -os /var/tmp [ 0.0] Opening the source -i disk /var/tmp/rhel-8.4.img [ 0.0] Creating an overlay to protect the source from being modified [ 0.1] Opening the overlay [ 4.8] Inspecting the overlay [ 9.9] Checking for sufficient free disk space in the guest [ 9.9] Estimating space required on target for each disk [ 9.9] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown device "vda". You may have to fix this entry manually after conversion. virt-v2v: This guest has virtio drivers installed. [ 48.8] Mapping filesystem data to avoid copying unused and blank areas [ 50.7] Closing the overlay [ 51.2] Assigning disks to buses [ 51.2] Checking if the guest needs BIOS or UEFI to boot [ 51.2] Initializing the target -o local -os /var/tmp [ 51.2] Copying disk 1/1 to /var/tmp/rhel-8.4-sda (raw) (100.00/100%) [ 58.3] Creating output metadata [ 58.3] Finishing off Now virt-v2v 2.0 doing just the conversion step: + ./convert/helper-v2v-convert /tmp/v2v [ 0.0] Opening the source [ 4.8] Inspecting the source [ 11.6] Checking for sufficient free disk space in the guest [ 11.6] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM helper-v2v-convert: warning: /files/boot/grub2/device.map/hd0 references unknown device "vda". You may have to fix this entry manually after conversion. helper-v2v-convert: This guest has virtio drivers installed. [ 50.7] Mapping filesystem data to avoid copying unused and blank areas [ 53.9] Closing the overlay [ 54.5] Assigning disks to buses [ 54.5] Checking if the guest needs BIOS or UEFI to boot [ 54.5] Creating output metadata [ 54.5] Finishing off You can see that it's about 2-3 seconds slower overall. You have to add to this: * About 2 seconds taken to start the input and output nbdkit instances. That's because we start the processes and then wait for them to write PID files, in a loop which sleeps for 1 second each iteration. We could be more intelligent here to reduce the sleep time, and we could also start the two sides in parallel. * nbdcopy takes 7.1 seconds, which is precisely the same time that qemu-img convert took (which is quite remarkable considering it's doing more work). We also hope to make nbdcopy much faster. So you can see what filters and options I'm using, the two nbdkit commands are: 3008833 pts/3 S+ 0:00 nbdkit --exit-with-parent --foreground --newstyle --pidfile /run/user/1000/v2vnbdkit.Fzy4p3/nbdkit1.pid --unix /tmp/v2v/in0 --threads 16 -D nbdkit.backend.datapath=0 --filter cow file file=/var/tmp/rhel-8.4.img fadvise=sequential cache=none 3008863 pts/3 S+ 0:00 nbdkit --exit-with-parent --foreground --newstyle --pidfile /run/user/1000/v2vnbdkit.pXctgg/nbdkit1.pid --unix /tmp/v2v/out0 --threads 16 -D nbdkit.backend.datapath=0 file file=/var/tmp/rhel-8.4-sda fadvise=sequential cache=none I'm not yet using nbdkit-cache-filter. Old virt-v2v uses copy-on-read caching. I'm also not yet using readahead, because nbdkit-readahead-filter is broken and needs a rewrite. Old virt-v2v uses readahead and adjusts it dynamically. The main aim of the virt-v2v split is not necessarily to make single instances faster (although we hope because of optimizations in nbdcopy to do that), but to make the overall system more flexible and take a holistic overview of what's happening on the system. The shell script I'm using to automate the new v2v is attached (because I haven't yet converted the command line parsing code). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

1 0

[PATCH libnbd] info: Add --map --totals sub-mode to display summary
by Richard W.M. Jones 28 Jun '21

28 Jun '21

This adds an nbdinfo sub-option --map --totals to display the total of each type of mapping: $ nbdkit -r file fedora-33.img --run 'nbdinfo --map --totals $uri' 1226113024 19.0 0 data 5216337920 81.0 3 hole,zero I need this kind of functionality for virt-v2v in order to estimate how much space is required on the target, although for virt-v2v I will probably use the API directly. Nevertheless it seems like a useful feature and the implementation does not complicate nbdinfo. It can also be used for other map types, eg. for showing how much of the dirty-bitmap is dirty. The implementation could be done in a single pass although for the vast majority of use cases I doubt that would be any faster since the table easily fits in L1 cache, and it would require a complex data structure. But it's worth noting that a malicious server could try to make nbdinfo use O(n^2) time (with a very small constant). Rich.

2 3

[PATCH libnbd] Avoid divide by zero
by Richard W.M. Jones 26 Jun '21

26 Jun '21

To be squashed into the previously posted patch: https://listman.redhat.com/archives/libguestfs/2021-June/msg00189.html

1 1

Re: [Libguestfs] oVirt import from VMware
by Richard W.M. Jones 23 Jun '21

23 Jun '21

Thanks for sending the disk image across. With latest libguestfs and guestfs-tools I am able to inspect the image fine: $ rpm -q guestfs-tools libguestfs guestfs-tools-1.47.2-1.fc35.x86_64 libguestfs-1.45.6-6.fc35.x86_64 $ tar xf SoftdriveTest.ova SoftdriveTest2-disk1.vmdk $ virt-inspector -a SoftdriveTest2-disk1.vmdk <?xml version="1.0"?> <operatingsystems> <operatingsystem> <root>/dev/sda2</root> <name>windows</name> <arch>x86_64</arch> <distro>windows</distro> <product_name>Windows 10 Home</product_name> <product_variant>Client</product_variant> <major_version>10</major_version> <minor_version>0</minor_version> <windows_systemroot>/Windows</windows_systemroot> <windows_current_control_set>ControlSet001</windows_current_control_set> <osinfo>win10</osinfo> <mountpoints> <mountpoint dev="/dev/sda2">/</mountpoint> </mountpoints> ... Recent virt-v2v gets further, but hits an issue because the guest is using Fast Restart: $ rpm -q virt-v2v virt-v2v-1.45.1-1.fc35.x86_64 $ virt-v2v -i ova SoftdriveTest.ova -o null [ 0.0] Opening the source -i ova SoftdriveTest.ova [ 166.6] Creating an overlay to protect the source from being modified [ 166.7] Opening the overlay [ 173.0] Inspecting the overlay virt-v2v: error: filesystem was mounted read-only, even though we asked for it to be mounted read-write. This usually means that the filesystem was not cleanly unmounted. Possible causes include trying to convert a guest which is running, or using Windows Hibernation or Fast Restart. (https://libguestfs.org/virt-v2v.1.html#windows-8-fast-startup-is-incompatib…) Looking at the detailed log, the inspection step was fine. Therefore I think it should work fine if Fast Restart was disabled before conversion. I went back to the old version of libguestfs you're using (1.40) virt-inspector shows pretty similar output to your virt-v2v log. However when I looked closer I found that guestfish cannot mount the /dev/sda2 partition from this disk at all. I'm not at all clear why. I guess it might be fast restart or something else (too old ntfs-3g?) Anyway since it works with recent libguestfs/virt-v2v, I would suggest upgrading to a later version and disabling Fast Restart in Windows guests before conversion, which together should solve the problem. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

2 9