May 2024 - Libguestfs - Libguestfs List Archives

[PATCH nbdkit 0/3] Add tls-* parameters to nbdkit URIs & more

by Richard W.M. Jones

I pushed some preliminary patches for this series already since they were all neutral refactorings. These three patches are the guts of it. They pass tests, I would push them ... but is this secure? I can't imagine a way that it's not secure, but maybe there's something about copying file locations from the nbdkit command line into the --run script which is problematic. Anyway ... The first patch modifies $uri in the --run script to add &tls-certificates=... or &tls-psk-file=... so that TLS connections work (with libnbd anyway, I didn't check qemu). The second patch adds these as variables as well. The third patch updates a test to use this. A final point is that this doesn't work well with PSK. PSK has an implicit username (passed in the URI as nbds://user@...). The same username has to appear in the PSK file. We don't set a username at all right now, so libnbd uses the LOGNAME, which may or may not be what you want. The way to fix this would be to add a --tls-username parameter to nbdkit, but it would only be useful for URIs and might confuse things otherwise (eg. it could be used with --tls-psk but would have no apparent effect, thus confusing users). Rich.

4 days

2
7
0 / 0

what size should NBD_OPT_INFO report?

by Eric Blake

I noticed during integration testing that nbd-server blindly reports a size of 0 for all NBD_OPT_INFO requests, unless I pass a size argument on the command line to nbd-server. At first, I thought it was a side effect of me trying to use nbd-server on a block device (an LVM partition), as it is a common bug to rely on stat().st_size which only works for regular files (a block device has to use lseek(SEEK_END)); but then I noticed it happening when using nbd-server to serve regular files as well. I then turned to the source code, where I see that client->exportsize is set in just these places: commit_client() - exportsize = OFFT_MAX, then try setupexport() setupexport() - default to client->server->expected_size (if one was provided), further validating that actual size is large enough when actual size can be computed - if neither treefile or F_WAIT is set, compute actual size by opening one or more files and using size_autodetect() (which does the right thing for block devices, so my earlier thought about over-reliance on stat() was wrong) but these functions are only reached for NBD_OPT_EXPORT_NAME and NBD_OPT_GO, not NBD_OPT_INFO. The upshot is that for NBD_OPT_GO, there are some scenarios (treefile, F_WAIT) where nbd-server advertises a size of 9223372036854775807 (0x7fffffff_ffffffff) meaning unknown, but a size of 0 there is only possible if the file was successfully opened and really is zero bytes in length. Conversely, NBD_OPT_INFO is always advertising a size of 0, which means most of the time, the size changes between NBD_OPT_INFO and NBD_OPT_GO. For comparison, both nbdkit and qemu as an NBD server always advertise the same size for both NBD_OPT_INFO and NBD_OPT_GO (but it should also be noted that neither of these has the complexity of multifile like nbd-server). Should we tweak the NBD standard to recommend that a server advertise a size of OFFT_MAX for NBD_OPT_INFO if it is prohibitive to determine an actual size, rather than 0? Furthermore, is it worth adding code to make NBD_OPT_INFO try harder to provide a sensible value when possible (if expected_size was set, or if we are not multifile or F_WAIT, then a stat() is enough to get the size for serving a regular file; and if stat() says we have a block device, we can still try the open/lseek/close)? Also, is it worth trying to specify that since no known NBD servers allow exports with sizes larger than OFFT_MAX, a server MUST NOT report a size equal or larger than 9223372036854775808 (0x80000000_00000000)? There are definitely a few places that I could simplify in libnbd if we have a protocol guarantee that a valid export size will never have the most significant bit set, and therefore we don't have to worry about whether size is represented as a signed or unsigned value (which also implies that libnbd's function nbd_get_size() returning ssize_t instead of size_t is acceptable). But if we do that, then maybe 0x80000000_00000000 or even 0xffffffff_ffffffff would serve as a better recommended sentinel than 0x7fffffff_ffffffff for representing an indeterminate size. -- Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org

4 days, 1 hour

3
6
0 / 0

[PATCH v1] appliance: only wait for resolv.conf update if dhcpcd succeeded

by Olaf Hering

In case network was requested, but the host lacks both dhclient and dhcpcd, skip the loop which waits for a resolv.conf update. This reduces boot time by 10 seconds. Signed-off-by: Olaf Hering <olaf(a)aepfle.de> --- Not runtime tested. appliance/init | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/appliance/init b/appliance/init index cb200d5bf..128a3c20e 100755 --- a/appliance/init +++ b/appliance/init @@ -119,8 +119,7 @@ if test "$guestfs_network" = 1; then rm -f /etc/dhcp/dhclient-enter-hooks.d/resolved if dhclient --version >/dev/null 2>&1; then dhclient $iface - else - dhcpcd $iface + elif dhcpcd $iface; then # https://github.com/NetworkConfiguration/dhcpcd/issues/258 for i in `seq 0 10`; do if grep nameserver /etc/resolv.conf; then break; fi

4 days, 18 hours

2
1
0 / 0

nbdkit | Failed pipeline for master | d1c49dec

by GitLab

Pipeline #1289238295 has failed! Project: nbdkit ( https://gitlab.com/nbdkit/nbdkit ) Branch: master ( https://gitlab.com/nbdkit/nbdkit/-/commits/master ) Commit: d1c49dec ( https://gitlab.com/nbdkit/nbdkit/-/commit/d1c49dec559a2a930b27a19c4b4acb9... ) Commit Message: tests: Rework TLS tests to use nbdinfo instead ... Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #1289238295 ( https://gitlab.com/nbdkit/nbdkit/-/pipelines/1289238295 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 2 failed jobs. Job #6841700897 ( https://gitlab.com/nbdkit/nbdkit/-/jobs/6841700897/raw ) Stage: builds Name: x86_64-almalinux-8-clang Job #6841700893 ( https://gitlab.com/nbdkit/nbdkit/-/jobs/6841700893/raw ) Stage: builds Name: x86_64-almalinux-8 -- You're receiving this email because of your account on gitlab.com.

6 days, 13 hours

1
0
0 / 0

nbdkit | Failed pipeline for master | fcf5073d

by GitLab

Pipeline #1288426020 has failed! Project: nbdkit ( https://gitlab.com/nbdkit/nbdkit ) Branch: master ( https://gitlab.com/nbdkit/nbdkit/-/commits/master ) Commit: fcf5073d ( https://gitlab.com/nbdkit/nbdkit/-/commit/fcf5073d1adb1e92e02cbfca9c1ca3d... ) Commit Message: server: Add --print-uri option Add a --print-u... Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #1288426020 ( https://gitlab.com/nbdkit/nbdkit/-/pipelines/1288426020 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 1 failed job. Job #6836613062 ( https://gitlab.com/nbdkit/nbdkit/-/jobs/6836613062/raw ) Stage: builds Name: aarch64-macos-14 -- You're receiving this email because of your account on gitlab.com.

6 days, 22 hours

1
0
0 / 0

downloaded file does not have correct content

by tommy.hensschen＠fau.de

Hallo Zusammen! Ich hoffe ich bin hier richtig und es ist okay, dass ich meine Frage hier stelle. Ich nutze GuestFS zur Zeit um aus einem QEMU-Abbild (qcow2) Dateien herunterzuladen und zu listen. Dafür nutze ich die python-bindings. Das herunterladen klappt auch soweit, allerdings haben die Dateien oft nicht den richtigen Inhalt. Ein Beispiel: - VM startet - herunterladen von Dateien z.B. "/Windows/System32/config/SOFTWARE" - Sonstige Dinge in der VM ausführen - VM herunterfahren - erneutes Herunterladen der gleichen Dateien nur in ein anderes Verzeichnis Oft sind die Dateien, die zu unterschiedlichen Zeiten heruntergeladen wurden, jedoch identisch. Wohl bemerkt obwohl die VM heruntergefahren ist. Wenn ich nun jedoch im Nachgang nochmal eine neue unabhängige Instanz meiner Python Klasse starte, werden die Dateien mit dem richtigem Inhalt heruntergeladen. Da ich irgendwelche Cacheprobleme im Verdacht habe, habe ich für das finale Herunterladen die Partition auch nochmal unmounted und wieder gemountet. Kann mir das jemand erklären oder mache ich was falsch? Wäre für Hinweise echt dankbar! Version: 1.52.0 stable Greetings Tommy

1 week

1
0
0 / 0

[nbdkit PATCH] tests: More debug output in test-blocksize-sharding.sh

by Eric Blake

We got a report of a test-blocksize-sharding.sh failure on ArchLinux [1], with a log pointing to the sanity check taking longer than expected. Since there were insufficient breadcrumbs to make it easy to see what went wrong, enhance the test to track more points in time and avoid confusing output ordering. Also, give the test a bit more leeway (3.5 seconds for the sanity check is still okay). [1] https://0x0.st/X8TI.log CC: solstice.dhiver(a)gmail.com Signed-off-by: Eric Blake <eblake(a)redhat.com> --- Formalizing the previous discussion, and pushing this now. tests/test-blocksize-sharding.sh | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/tests/test-blocksize-sharding.sh b/tests/test-blocksize-sharding.sh index 7e926ad1..9db5b220 100755 --- a/tests/test-blocksize-sharding.sh +++ b/tests/test-blocksize-sharding.sh @@ -80,9 +80,13 @@ witness = os.getenv("witness") def touch(path): open(path, "a").close() +def progress(prefix, start): + t = time.time() - start + print("%s: %g" % (prefix, t), flush=True) + # First pass: check that two aligned operations work in parallel # Total time should be closer to 2 seconds, rather than 4 if serialized -print("sanity check") +print("sanity check", flush=True) ba1 = bytearray(b"1"*16) ba2 = bytearray(b"2"*16) buf1 = nbd.Buffer.from_bytearray(ba1) @@ -91,21 +95,24 @@ touch(witness) start_t = time.time() h.aio_pwrite(buf1, 0) h.aio_pwrite(buf2, 0) +progress("after writes", start_t) while h.aio_in_flight() > 0: h.poll(-1) + progress("after poll", start_t) end_t = time.time() os.unlink(witness) out = h.pread(16,0) -print(out) +progress("after read", start_t) +print(out, flush=True) t = end_t - start_t -print(t) +print("elapsed write time: %g" % t, flush=True) assert out in [b"1"*16, b"2"*16] -assert t >= 2 and t <= 3 +assert 2.0 <= t < 4.0 # Next pass: try to kick off unaligned first -print("unaligned first") +print("unaligned first", flush=True) h.zero(16, 0) ba3 = bytearray(b"3"*12) ba4 = bytearray(b"4"*16) @@ -122,14 +129,14 @@ end_t = time.time() os.unlink(witness) out = h.pread(16,0) -print(out) +print(out, flush=True) t = end_t - start_t -print(t) +print(t, flush=True) assert out in [b"4"*4 + b"3"*12, b"4"*16] -assert t >= 8 +assert t >= 8.0 # Next pass: try to kick off aligned first -print("aligned first") +print("aligned first", flush=True) ba5 = bytearray(b"5"*16) ba6 = bytearray(b"6"*12) buf5 = nbd.Buffer.from_bytearray(ba5) @@ -146,11 +153,11 @@ end_t = time.time() os.unlink(witness) out = h.pread(16,0) -print(out) +print(out, flush=True) t = end_t - start_t -print(t) +print(t, flush=True) assert out in [b"5"*4 + b"6"*12, b"5"*16] -assert t >= 8 +assert t >= 8.0 ' # Now run everything -- 2.45.0

1 week, 3 days

1
0
0 / 0

nbdkit test-blocksize-sharding.sh failing

by Richard W.M. Jones

Hi Eric, ----- Forwarded message from notify(a)aur.archlinux.org ----- solsticedhiver [1] added the following comment to nbdkit [2]: There is still one test failing now test-blocksize-sharding.sh. I don't know if it is expected or not The full test-suite.log https://0x0.st/X8TI.log [1] https://aur.archlinux.org/account/solsticedhiver/ [2] https://aur.archlinux.org/pkgbase/nbdkit/ ----- End forwarded message ----- I've attached the fragment of the failure log that they sent. However I don't understand the test enough to see where the error happens. Additionally I think there's a problem with the various 'print'- statements in the Python code. They probably need to flush stdout otherwise they don't appear coherent with error messages. Something like this thoroughout: -print("sanity check") +print("sanity check", flush=True) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit

1 week, 4 days

2
2
0 / 0

libnbd | Failed pipeline for master | 139d56a5

by GitLab

Pipeline #1281932671 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: 139d56a5 ( https://gitlab.com/nbdkit/libnbd/-/commit/139d56a57f5f4fd5d0641966ade4f62... ) Commit Message: Version 1.20.1. Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #1281932671 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/1281932671 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 3 failed jobs. Job #6798867170 ( https://gitlab.com/nbdkit/libnbd/-/jobs/6798867170/raw ) Stage: builds Name: x86_64-alpine-edge Job #6798867326 ( https://gitlab.com/nbdkit/libnbd/-/jobs/6798867326/raw ) Stage: builds Name: aarch64-macos-13 Job #6798867315 ( https://gitlab.com/nbdkit/libnbd/-/jobs/6798867315/raw ) Stage: builds Name: x86_64-freebsd-14 -- You're receiving this email because of your account on gitlab.com.

1 week, 5 days

1
0
0 / 0

libnbd | Failed pipeline for master | 7ec39ce1

by GitLab

Pipeline #1281916695 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: 7ec39ce1 ( https://gitlab.com/nbdkit/libnbd/-/commit/7ec39ce1fba9a5e555e202e8f013001... ) Commit Message: Merge branch '2024-code-tests' into 'master' t... Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #1281916695 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/1281916695 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 1 failed job. Job #6798752052 ( https://gitlab.com/nbdkit/libnbd/-/jobs/6798752052/raw ) Stage: builds Name: aarch64-macos-13 -- You're receiving this email because of your account on gitlab.com.

1 week, 5 days

1
0
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Libguestfs May 2024