nbdcpy: from scratch nbdcopy using io_uring
by Abhay Raj Singh
In a previous email, Mr. Jones suggested, writing a test implementation of
nbdcopy from scratch. First of all, thank you, Mr. Jones, as I got a better
perspective of the problem and the solution.
I have almost written the bare-bones implementation for handling
NBD-related stuff on which I will base the io_uring core piece by piece. It
is written in C++, as I am more comfortable with it. But, made it easily
translatable to C.
I have been working on a solution and have documented the current approach,
it would be great if you have a look at it and let me know what you think.
Approach document:
https://gitlab.com/rathod-sahaab/nbdcpy/-/blob/dev/docs/SOLUTION.md
Thanks and regards,
Abhay
3 years, 2 months
[PATCH] point users to Libera Chat rather than FreeNode
by Daniel P. Berrangé
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
docs/guestfs-faq.pod | 2 +-
website/index.html.in | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/docs/guestfs-faq.pod b/docs/guestfs-faq.pod
index bea609591..15b1247b0 100644
--- a/docs/guestfs-faq.pod
+++ b/docs/guestfs-faq.pod
@@ -108,7 +108,7 @@ There is a mailing list, mainly for development, but users are also
welcome to ask questions about libguestfs and the virt tools:
L<https://www.redhat.com/mailman/listinfo/libguestfs>
-You can also talk to us on IRC channel C<#libguestfs> on FreeNode.
+You can also talk to us on IRC channel C<#guestfs> on Libera Chat.
We're not always around, so please stay in the channel after asking
your question and someone will get back to you.
diff --git a/website/index.html.in b/website/index.html.in
index f469c5eeb..7453129d6 100644
--- a/website/index.html.in
+++ b/website/index.html.in
@@ -55,8 +55,8 @@ guestfish --ro -i -a disk.img
<p>
Join us on
the <a href="http://www.redhat.com/mailman/listinfo/libguestfs">libguestfs
-mailing list</a>, or on IRC channel <code>#libguestfs</code>
-on <a href="http://freenode.net/">FreeNode</a>.
+mailing list</a>, or on IRC channel <code>#guestfs</code>
+on <a href="https://libera.chat/">Libera Chat</a>.
</p>
</div>
--
2.31.1
3 years, 3 months
Figuring out some failing tests for libnbd
by Martin Kletzander
I am preparing more patches for CI to run check-valgrind and fix ongoing
errors but there are two issues I can not identify the reason why they
are failing.
- On debian-10 the info/info-can.sh started failing and the error
message is just one of those I saw earlier in other places:
libnbd: debug: nbd1: nbd_opt_abort: leave: error="nbd_opt_abort:
invalid state: READY: the handle must be negotiating: Invalid
argument"
- On Fedora rawhide I hit a random issue where a port in a URI was
translated to its name and looking at the code I can not find how this
could have happened. Until this is fixed the test suite is unreliable
and notification fatigue will cause everyone to start ignoring any
future failures.
/builds/nertpinx/libnbd/tests/.libs/aio-connect: actual URI
nbd://127.0.0.1:altova-lm/ != expected URI nbd://127.0.0.1:35355/
- Both openSUSE builds are failing to run check-valgrind and it looks
like it might be unrelated to libnbd, although it would be nice for
someone else to confirm that. For now I have disabled check-valgrind
on those platforms in my branch.
- Similarly to openSUSE Ubuntu 20.04 fails in valgrind tests, but
somewhere down the GnuTLS rabbit hole, which I presume is unrelated
too, so I disabled check-valgrind on that one as well.
I will send the patches once they are cleaned up, but I wanted to let
everyone know what the current status is because eliminating all random
issues is essential to properly consuming CI results.
Thanks,
Martin
3 years, 4 months
[PATCH libnbd] lib/uri.c: nbd_get_uri: Do not translate port name into service
by Richard W.M. Jones
Because I forgot to use the NI_NUMERICSERV flag, getnameinfo would
translate the port number into a service name (when possible) using
/etc/services. This is not really desirable since raw port numbers
are more portable and descriptive. This also caused occasional test
failures when pick-a-port happened to pick a port number which
coincided with a service name.
This commit also adds a test. Previous to the fix, the test failed with:
$ ./run tests/aio-connect-port
tests/.libs/aio-connect-port: actual URI nbd://127.0.0.1:sieve-filter/ != expected URI nbd://127.0.0.1:2000/
This test cannot be run as part of the test suite because it requires
a fixed port number.
See also
https://listman.redhat.com/archives/libguestfs/2021-June/msg00205.html
Reported-by: Martin Kletzander
---
.gitignore | 1 +
lib/uri.c | 3 +-
tests/Makefile.am | 9 +++
tests/aio-connect-port.c | 128 +++++++++++++++++++++++++++++++++++++++
4 files changed, 140 insertions(+), 1 deletion(-)
diff --git a/.gitignore b/.gitignore
index 865b689..602438d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -155,6 +155,7 @@ Makefile.in
/stamp-h1
/test-driver
/tests/aio-connect
+/tests/aio-connect-port
/tests/aio-parallel
/tests/aio-parallel-load
/tests/aio-parallel-load-tls
diff --git a/lib/uri.c b/lib/uri.c
index 9b97e46..bcecbad 100644
--- a/lib/uri.c
+++ b/lib/uri.c
@@ -422,7 +422,8 @@ nbd_unlocked_get_uri (struct nbd_handle *h)
uri.scheme = using_tls ? "nbds" : "nbd";
err = getnameinfo ((struct sockaddr *) &h->connaddr, h->connaddrlen,
- host, sizeof host, serv, sizeof serv, NI_NUMERICHOST);
+ host, sizeof host, serv, sizeof serv,
+ NI_NUMERICHOST | NI_NUMERICSERV);
if (err != 0) {
set_error (0, "getnameinfo: %s", gai_strerror (err));
goto out;
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 09de079..aad5513 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -203,6 +203,7 @@ check_PROGRAMS += \
connect-tcp \
connect-tcp6 \
aio-connect \
+ aio-connect-port \
aio-parallel \
aio-parallel-load \
synch-parallel \
@@ -252,6 +253,9 @@ TESTS += \
closure-lifetimes \
$(NULL)
+# This test is compiled but not run because it requires a fixed port:
+# aio-connect-port
+
errors_SOURCES = errors.c
errors_CPPFLAGS = -I$(top_srcdir)/include
errors_CFLAGS = $(WARNINGS_CFLAGS)
@@ -489,6 +493,11 @@ aio_connect_CPPFLAGS = -I$(top_srcdir)/include
aio_connect_CFLAGS = $(WARNINGS_CFLAGS)
aio_connect_LDADD = $(top_builddir)/lib/libnbd.la
+aio_connect_port_SOURCES = aio-connect-port.c
+aio_connect_port_CPPFLAGS = -I$(top_srcdir)/include
+aio_connect_port_CFLAGS = $(WARNINGS_CFLAGS)
+aio_connect_port_LDADD = $(top_builddir)/lib/libnbd.la
+
aio_parallel_SOURCES = aio-parallel.c
aio_parallel_CPPFLAGS = \
-I$(top_srcdir)/include \
diff --git a/tests/aio-connect-port.c b/tests/aio-connect-port.c
new file mode 100644
index 0000000..5f9613e
--- /dev/null
+++ b/tests/aio-connect-port.c
@@ -0,0 +1,128 @@
+/* NBD client library in userspace
+ * Copyright (C) 2013-2021 Red Hat Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/* Test connecting to a fixed IPv4 TCP port (2000) using
+ * nbd_aio_connect. This test is compiled but not normally run
+ * because of the requirement for the fixed port number.
+ *
+ * See also:
+ * https://listman.redhat.com/archives/libguestfs/2021-June/msg00205.html
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <arpa/inet.h>
+
+#include <libnbd.h>
+
+#define PIDFILE "aio-connect-port.pid"
+
+int
+main (int argc, char *argv[])
+{
+ struct nbd_handle *nbd;
+ int port = 2000;
+ const char *port_str = "2000";
+ pid_t pid;
+ size_t i;
+ struct sockaddr_in addr;
+ char *actual_uri, *expected_uri;
+
+ unlink (PIDFILE);
+
+ pid = fork ();
+ if (pid == -1) {
+ perror ("fork");
+ exit (EXIT_FAILURE);
+ }
+ if (pid == 0) {
+ execlp ("nbdkit",
+ "nbdkit", "-f", "-p", port_str, "-P", PIDFILE,
+ "--exit-with-parent", "null", NULL);
+ perror ("nbdkit");
+ _exit (EXIT_FAILURE);
+ }
+
+ /* Wait for nbdkit to start listening. */
+ for (i = 0; i < 60; ++i) {
+ if (access (PIDFILE, F_OK) == 0)
+ break;
+ sleep (1);
+ }
+ unlink (PIDFILE);
+
+ nbd = nbd_create ();
+ if (nbd == NULL) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+ if (nbd_supports_uri (nbd) != 1) {
+ fprintf (stderr, "skip: compiled without URI support\n");
+ exit (77);
+ }
+
+ addr.sin_family = AF_INET;
+ addr.sin_addr.s_addr = htonl (INADDR_LOOPBACK);
+ addr.sin_port = htons (port);
+
+ if (nbd_aio_connect (nbd, (struct sockaddr *) &addr, sizeof addr) == -1) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ /* Wait until we have connected. */
+ while (!nbd_aio_is_ready (nbd)) {
+ if (nbd_poll (nbd, -1) == -1) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+ }
+
+ /* libnbd should be able to construct a URI for this connection. */
+ if (asprintf (&expected_uri, "nbd://127.0.0.1:%s/", port_str) == -1) {
+ perror ("asprintf");
+ exit (EXIT_FAILURE);
+ }
+ actual_uri = nbd_get_uri (nbd);
+ if (actual_uri == NULL) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+ if (strcmp (actual_uri, expected_uri) != 0) {
+ fprintf (stderr, "%s: actual URI %s != expected URI %s\n",
+ argv[0], actual_uri, expected_uri);
+ exit (EXIT_FAILURE);
+ }
+ free (actual_uri);
+ free (expected_uri);
+
+ if (nbd_shutdown (nbd, 0) == -1) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ nbd_close (nbd);
+ exit (EXIT_SUCCESS);
+}
--
2.32.0
3 years, 4 months
[PATCH libnbd] python: Implement nbd.aio_connect for AF_UNIX
by Richard W.M. Jones
This call previously just called abort(). Implement it for Unix
domain sockets (the easy case). Implementing it for AF_INET and
AF_INET6 is more complicated so that is left as a to-do.
Note also that implementing this fully for Python is a bit pointless.
It would be easier for a Python program to call
nbd.aio_connect_tcp(host, port) instead of calling
nbd.aio_connect((host, port)). Both cases would do hostname lookups
but the former is already implemented.
---
generator/Python.ml | 17 +++++----
python/Makefile.am | 6 ++-
python/python-aio-connect-unix.sh | 36 ++++++++++++++++++
python/utils.c | 63 +++++++++++++++++++++++++++++++
4 files changed, 114 insertions(+), 8 deletions(-)
diff --git a/generator/Python.ml b/generator/Python.ml
index 25f92d7..bcfd4bf 100644
--- a/generator/Python.ml
+++ b/generator/Python.ml
@@ -41,6 +41,8 @@ struct py_aio_buffer {
extern char **nbd_internal_py_get_string_list (PyObject *);
extern void nbd_internal_py_free_string_list (char **);
+extern int nbd_internal_py_get_sockaddr (PyObject *,
+ struct sockaddr_storage *, socklen_t *);
extern struct py_aio_buffer *nbd_internal_py_get_aio_buffer (PyObject *);
static inline struct nbd_handle *
@@ -300,10 +302,9 @@ let print_python_binding name { args; optargs; ret; may_set_error } =
| SizeT n ->
pr " Py_ssize_t %s;\n" n
| SockAddrAndLen (n, _) ->
- pr " /* XXX Complicated - Python uses a tuple of different\n";
- pr " * lengths for the different socket types.\n";
- pr " */\n";
- pr " PyObject *%s;\n" n
+ pr " PyObject *%s;\n" n;
+ pr " struct sockaddr_storage %s_sa;\n" n;
+ pr " socklen_t %s_len;\n" n;
| String n -> pr " const char *%s;\n" n
| StringList n ->
pr " PyObject *py_%s;\n" n;
@@ -440,8 +441,10 @@ let print_python_binding name { args; optargs; ret; may_set_error } =
pr " %s = PyBytes_AS_STRING (py_%s);\n" n n;
pr " assert (%s != NULL);\n" n
| SizeT n -> ()
- | SockAddrAndLen _ ->
- pr " abort (); /* XXX SockAddrAndLen not implemented */\n";
+ | SockAddrAndLen (n, _) ->
+ pr " if (nbd_internal_py_get_sockaddr (%s, &%s_sa, &%s_len) == -1)\n"
+ n n n;
+ pr " goto out;\n"
| String _ -> ()
| StringList n ->
pr " %s = nbd_internal_py_get_string_list (py_%s);\n" n n;
@@ -468,7 +471,7 @@ let print_python_binding name { args; optargs; ret; may_set_error } =
| Int64 n -> pr ", %s_i64" n
| Path n -> pr ", %s" n
| SizeT n -> pr ", (size_t)%s" n
- | SockAddrAndLen (n, _) -> pr ", /* XXX */ (void *) %s, 0" n
+ | SockAddrAndLen (n, _) -> pr ", (struct sockaddr *) &%s_sa, %s_len" n n
| String n -> pr ", %s" n
| StringList n -> pr ", %s" n
| UInt n | UIntPtr n -> pr ", %s" n
diff --git a/python/Makefile.am b/python/Makefile.am
index 718f891..3779410 100644
--- a/python/Makefile.am
+++ b/python/Makefile.am
@@ -81,9 +81,13 @@ TESTS_ENVIRONMENT = \
LIBNBD_DEBUG=1 \
MALLOC_CHECK_=1 \
MALLOC_PERTURB_=$(shell bash -c 'echo $$(( 1 + (RANDOM & 255) ))') \
+ PYTHON="$(PYTHON)" \
$(NULL)
LOG_COMPILER = $(top_builddir)/run
-TESTS += run-python-tests
+TESTS += \
+ python-aio-connect-unix.sh \
+ run-python-tests \
+ $(NULL)
endif HAVE_NBDKIT
diff --git a/python/python-aio-connect-unix.sh b/python/python-aio-connect-unix.sh
new file mode 100755
index 0000000..7be0f87
--- /dev/null
+++ b/python/python-aio-connect-unix.sh
@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# nbd client library in userspace
+# Copyright (C) 2020-2021 Red Hat Inc.
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+. ../tests/functions.sh
+
+set -e
+set -x
+
+requires nbdkit --version
+requires $PYTHON --version
+
+nbdkit -U - null --run '$PYTHON -c "
+import nbd
+import sys
+h = nbd.NBD()
+addr = sys.argv[1]
+h.aio_connect(addr)
+while h.aio_is_connecting():
+ h.poll(1)
+assert h.aio_is_ready()
+" "$unixsocket"'
diff --git a/python/utils.c b/python/utils.c
index 0e3164c..37f0c55 100644
--- a/python/utils.c
+++ b/python/utils.c
@@ -26,6 +26,8 @@
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
+#include <sys/socket.h>
+#include <sys/un.h>
#include <libnbd.h>
@@ -92,3 +94,64 @@ nbd_internal_py_free_string_list (char **argv)
free (argv[i]);
free (argv);
}
+
+/* Convert a Python object into a struct sockaddr, according to the
+ * general rules described here:
+ * https://docs.python.org/3/library/socket.html
+ *
+ * There is a function in cpython called getsockaddrarg which roughly
+ * does the same thing, but in cpython they know the socket family
+ * already (which we do not). In any case that function cannot be
+ * called directly.
+ */
+int
+nbd_internal_py_get_sockaddr (PyObject *addr,
+ struct sockaddr_storage *ss, socklen_t *len)
+{
+ memset (ss, 0, sizeof *ss);
+
+ if (PyUnicode_Check (addr)) { /* AF_UNIX */
+ struct sockaddr_un *sun = (struct sockaddr_un *)ss;
+ const char *unixsocket;
+ size_t namelen;
+
+ sun->sun_family = AF_UNIX;
+ unixsocket = PyUnicode_AsUTF8 (addr);
+ if (!unixsocket)
+ goto err;
+ namelen = strlen (unixsocket);
+ if (namelen > sizeof sun->sun_path) {
+ PyErr_SetString (PyExc_RuntimeError,
+ "get_sockaddr: Unix domain socket name too long");
+ return -1;
+ }
+ memcpy (sun->sun_path, unixsocket, namelen);
+ *len = sizeof *sun;
+ return 0;
+ }
+
+#if 0
+ else if (PyTuple_Check (addr)) {
+ Py_ssize_t n = PyTuple_Size (addr);
+
+ switch (n) {
+ case 2: /* AF_INET */
+ /* XXX TODO */
+ break;
+
+ case 4: /* AF_INET6 */
+ /* XXX TODO */
+ break;
+
+ default:
+ goto err;
+ }
+ }
+#endif
+
+ else {
+ err:
+ PyErr_SetString (PyExc_TypeError, "get_sockaddr: unknown address type");
+ return -1;
+ }
+}
--
2.32.0
3 years, 4 months
[libnbd PATCH 0/3] CI updates
by Martin Kletzander
Some fixes plus check-valgrind enablement.
Martin Kletzander (3):
qemu-storage-daemon 5.2.0 is still broken
fuse: move check-valgrind out from condition
ci: Rework the build script to run check-valgrind properly
interop/interop-qemu-storage-daemon.sh | 3 +-
.gitlab-ci.yml | 3 +
ci/build.sh | 96 +++++++++++---------
ci/cirrus/freebsd-12.vars | 4 +-
ci/cirrus/freebsd-13.vars | 4 +-
ci/cirrus/freebsd-current.vars | 4 +-
ci/cirrus/macos-11.vars | 5 +-
ci/containers/centos-8.Dockerfile | 5 +-
ci/containers/centos-stream-8.Dockerfile | 5 +-
ci/containers/debian-10.Dockerfile | 5 +-
ci/containers/debian-sid.Dockerfile | 5 +-
ci/containers/fedora-33.Dockerfile | 7 +-
ci/containers/fedora-34.Dockerfile | 7 +-
ci/containers/fedora-rawhide.Dockerfile | 7 +-
ci/containers/opensuse-leap-152.Dockerfile | 5 +-
ci/containers/opensuse-tumbleweed.Dockerfile | 5 +-
ci/containers/ubuntu-1804.Dockerfile | 5 +-
ci/containers/ubuntu-2004.Dockerfile | 5 +-
fuse/Makefile.am | 4 +-
19 files changed, 102 insertions(+), 82 deletions(-)
--
2.32.0
3 years, 4 months
Virt-v2v split performance
by Richard W.M. Jones
[Some random mumblings that I want to put on record ...]
The new "modular virt-v2v" (which I'll probably release as "virt-v2v
2.0" even though it's not really a rewrite) is in the state where it
can convert guests:
Alpha-quality code:
https://github.com/rwmjones/virt-v2v/commits/2021-virt-v2v-split
Original plan:
https://listman.redhat.com/archives/libguestfs/2020-November/msg00022.html
One thing of note is that it's slower in some places and maybe will be
faster in others. Because I'm using NBD disk pipelines for
everything, when we want to read or write to a local disk image we go
through nbdkit (instead of just accessing the local file as virt-v2v
would do). However the local file case is not very common in real
world usage (except for testing).
On the flip side, nbdcopy is faster than qemu-img convert when copying
between NBD source and target. In fact when copying from file to file
through 2 x nbdkits, nbdcopy is about the same speed as qemu-img convert
copying between local files.
Some actual timings. As a baseline here's original virt-v2v doing a
file to file conversion:
$ virt-v2v -i disk /var/tmp/rhel-8.4.img -o disk -os /var/tmp
[ 0.0] Opening the source -i disk /var/tmp/rhel-8.4.img
[ 0.0] Creating an overlay to protect the source from being modified
[ 0.1] Opening the overlay
[ 4.8] Inspecting the overlay
[ 9.9] Checking for sufficient free disk space in the guest
[ 9.9] Estimating space required on target for each disk
[ 9.9] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM
virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown
device "vda". You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[ 48.8] Mapping filesystem data to avoid copying unused and blank areas
[ 50.7] Closing the overlay
[ 51.2] Assigning disks to buses
[ 51.2] Checking if the guest needs BIOS or UEFI to boot
[ 51.2] Initializing the target -o local -os /var/tmp
[ 51.2] Copying disk 1/1 to /var/tmp/rhel-8.4-sda (raw)
(100.00/100%)
[ 58.3] Creating output metadata
[ 58.3] Finishing off
Now virt-v2v 2.0 doing just the conversion step:
+ ./convert/helper-v2v-convert /tmp/v2v
[ 0.0] Opening the source
[ 4.8] Inspecting the source
[ 11.6] Checking for sufficient free disk space in the guest
[ 11.6] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM
helper-v2v-convert: warning: /files/boot/grub2/device.map/hd0 references
unknown device "vda". You may have to fix this entry manually after
conversion.
helper-v2v-convert: This guest has virtio drivers installed.
[ 50.7] Mapping filesystem data to avoid copying unused and blank areas
[ 53.9] Closing the overlay
[ 54.5] Assigning disks to buses
[ 54.5] Checking if the guest needs BIOS or UEFI to boot
[ 54.5] Creating output metadata
[ 54.5] Finishing off
You can see that it's about 2-3 seconds slower overall.
You have to add to this:
* About 2 seconds taken to start the input and output nbdkit
instances. That's because we start the processes and then wait for
them to write PID files, in a loop which sleeps for 1 second each
iteration. We could be more intelligent here to reduce the sleep
time, and we could also start the two sides in parallel.
* nbdcopy takes 7.1 seconds, which is precisely the same time that
qemu-img convert took (which is quite remarkable considering it's
doing more work). We also hope to make nbdcopy much faster.
So you can see what filters and options I'm using, the two nbdkit
commands are:
3008833 pts/3 S+ 0:00 nbdkit --exit-with-parent --foreground --newstyle --pidfile /run/user/1000/v2vnbdkit.Fzy4p3/nbdkit1.pid --unix /tmp/v2v/in0 --threads 16 -D nbdkit.backend.datapath=0 --filter cow file file=/var/tmp/rhel-8.4.img fadvise=sequential cache=none
3008863 pts/3 S+ 0:00 nbdkit --exit-with-parent --foreground --newstyle --pidfile /run/user/1000/v2vnbdkit.pXctgg/nbdkit1.pid --unix /tmp/v2v/out0 --threads 16 -D nbdkit.backend.datapath=0 file file=/var/tmp/rhel-8.4-sda fadvise=sequential cache=none
I'm not yet using nbdkit-cache-filter. Old virt-v2v uses copy-on-read
caching. I'm also not yet using readahead, because nbdkit-readahead-filter
is broken and needs a rewrite. Old virt-v2v uses readahead and
adjusts it dynamically.
The main aim of the virt-v2v split is not necessarily to make single
instances faster (although we hope because of optimizations in nbdcopy
to do that), but to make the overall system more flexible and take a
holistic overview of what's happening on the system.
The shell script I'm using to automate the new v2v is attached
(because I haven't yet converted the command line parsing code).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3 years, 4 months
[PATCH libnbd] info: Add --map --totals sub-mode to display summary
by Richard W.M. Jones
This adds an nbdinfo sub-option --map --totals to display the total of
each type of mapping:
$ nbdkit -r file fedora-33.img --run 'nbdinfo --map --totals $uri'
1226113024 19.0 0 data
5216337920 81.0 3 hole,zero
I need this kind of functionality for virt-v2v in order to estimate
how much space is required on the target, although for virt-v2v I will
probably use the API directly. Nevertheless it seems like a useful
feature and the implementation does not complicate nbdinfo.
It can also be used for other map types, eg. for showing how much of
the dirty-bitmap is dirty.
The implementation could be done in a single pass although for the
vast majority of use cases I doubt that would be any faster since the
table easily fits in L1 cache, and it would require a complex data
structure. But it's worth noting that a malicious server could try to
make nbdinfo use O(n^2) time (with a very small constant).
Rich.
3 years, 4 months
Re: [Libguestfs] oVirt import from VMware
by Richard W.M. Jones
Thanks for sending the disk image across.
With latest libguestfs and guestfs-tools I am able to inspect the image fine:
$ rpm -q guestfs-tools libguestfs
guestfs-tools-1.47.2-1.fc35.x86_64
libguestfs-1.45.6-6.fc35.x86_64
$ tar xf SoftdriveTest.ova SoftdriveTest2-disk1.vmdk
$ virt-inspector -a SoftdriveTest2-disk1.vmdk
<?xml version="1.0"?>
<operatingsystems>
<operatingsystem>
<root>/dev/sda2</root>
<name>windows</name>
<arch>x86_64</arch>
<distro>windows</distro>
<product_name>Windows 10 Home</product_name>
<product_variant>Client</product_variant>
<major_version>10</major_version>
<minor_version>0</minor_version>
<windows_systemroot>/Windows</windows_systemroot>
<windows_current_control_set>ControlSet001</windows_current_control_set>
<osinfo>win10</osinfo>
<mountpoints>
<mountpoint dev="/dev/sda2">/</mountpoint>
</mountpoints>
...
Recent virt-v2v gets further, but hits an issue because the guest is
using Fast Restart:
$ rpm -q virt-v2v
virt-v2v-1.45.1-1.fc35.x86_64
$ virt-v2v -i ova SoftdriveTest.ova -o null
[ 0.0] Opening the source -i ova SoftdriveTest.ova
[ 166.6] Creating an overlay to protect the source from being modified
[ 166.7] Opening the overlay
[ 173.0] Inspecting the overlay
virt-v2v: error: filesystem was mounted read-only, even though we asked for
it to be mounted read-write. This usually means that the filesystem was
not cleanly unmounted. Possible causes include trying to convert a guest
which is running, or using Windows Hibernation or Fast Restart.
(https://libguestfs.org/virt-v2v.1.html#windows-8-fast-startup-is-incompat...)
Looking at the detailed log, the inspection step was fine. Therefore
I think it should work fine if Fast Restart was disabled before conversion.
I went back to the old version of libguestfs you're using (1.40)
virt-inspector shows pretty similar output to your virt-v2v log.
However when I looked closer I found that guestfish cannot mount the
/dev/sda2 partition from this disk at all. I'm not at all clear why.
I guess it might be fast restart or something else (too old ntfs-3g?)
Anyway since it works with recent libguestfs/virt-v2v, I would suggest
upgrading to a later version and disabling Fast Restart in Windows
guests before conversion, which together should solve the problem.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3 years, 5 months
