October 2021 - Libguestfs - libguestfs List Archives

Publishing the libnbd go module
by Nir Soffer 25 Oct '21

25 Oct '21

I'm playing with libnbd go module, planning to use it in a new command[1] The biggest obstacle for me is that the module is not published in a way that Go developers expect. The module is listed in: https://pkg.go.dev/github.com/libguestfs/libnbd/golang But the module actually lives in: https://github.com/libguestfs/libnbd/tree/master/golang/src/libguestfs.org/… So the pkg.go.dev page is broken, .e.g no there is no documation or license, and the suggested import is wrong. The module name is "libguestfs.org/libnbd". But if you try to use it, for example in the (improved) example from libnbd-golang.pod: $ cat test.go package main import "fmt" import "libguestfs.org/libnbd" func main() { h, err := libnbd.Create() if err != nil { panic(err) } defer h.Close() uri := "nbd://localhost" err = h.ConnectUri(uri) if err != nil { panic(err) } size, err := h.GetSize() if err != nil { panic(err) } fmt.Printf("size of %s = %d\n", uri, size) } $ go mod init example/test go: creating new go.mod: module example/test go: to add module requirements and sums: go mod tidy $ go mod tidy go: finding module for package libguestfs.org/libnbd example/test imports libguestfs.org/libnbd: cannot find module providing package libguestfs.org/libnbd: unrecognized import path "libguestfs.org/libnbd": reading https://libguestfs.org/libnbd?go-get=1: 404 Not Found If we use libguestfs.org, https://libguestfs.org/libnbd?go-get=1 should return the expected metadata instead of 404. But even if "libguestfs.org/libnbd" would work, we cannot use the module from the source since the source is missing the generated files (wrappers.go, binding.go, ...). It looks like the only ways to use the module are: - Vendor the go code from the tarball. I did not try this since I don't want to copy libnbd into source into my project. - Clone and build libnbd locally, and replace libguestfs.org with the path to your local source $ go mod edit -replace libguestfs.org/libnbd=../../src/libnbd/golang/src/libguestfs.org/libnbd $ go mod tidy go: found libguestfs.org/libnbd in libguestfs.org/libnbd v0.0.0-00010101000000-000000000000 $ cat go.mod module example/test go 1.16 replace libguestfs.org/libnbd => ../../src/libnbd/golang/src/libguestfs.org/libnbd require libguestfs.org/libnbd v0.0.0-00010101000000-000000000000 But the version is wrong - it should be v1.10.0. I think the issue is missing tag: https://golang.org/ref/mod#vcs-version If a module is defined in a subdirectory within the repository, that is, the module subdirectory portion of the module path is not empty, then each tag name must be prefixed with the module subdirectory, followed by a slash. For example, the module golang.org/x/tools/gopls is defined in the gopls subdirectory of the repository with root path golang.org/x/tools. The version v0.4.0 of that module must have the tag named gopls/v0.4.0 in that repository. So the linbd project needs a tag like: golang/src/libguestfs.org/libnbd/v1.10.0 Removing the "src/libguestfs.org" directories will clean things up: golang/libnbd/v1.10.0 I hope we can solve this issue. Making the go binding easy to use for developers is important for making libnbd more popular in the Go community. [1] https://gerrit.ovirt.org/c/ovirt-imageio/+/117277 Nir

5 10

[PATCH v3 0/5] qapi: Add feature flags to enum members
by Markus Armbruster 25 Oct '21

25 Oct '21

PATCH 1+2 add feature flags to enum members. Awkward due to an introspection design mistake; see PATCH 1 for details. PATCH 3+4 implement policy deprecated-input={reject,crash} for enum values. Policy deprecated-output=hide is not implemented, because we can't hide a value without hiding the entire member, which is almost certainly more than the requester of this policy bargained for. Perhaps we want a new policy deprecated-output=hide-or-else-crash to help us catch unwanted use of deprecated enum values. Perhaps we want deprecated-output=hide to behave that way together with deprecated-input=crash. Or even always. Thoughts? PATCH 5 puts the new feature flags to use. It's RFC because it makes sense only on top of Vladimir's deprecation of drive-backup. See its commit message for a reference. I prefer to commit new features together with a use outside tests/. PATCH 5 adds such a use, but it's RFC, because it depends on Vladimir's work. Perhaps another use pops up. I can delay this work in the hope of a use becoming ready, but the feature flags work I have in the pipeline will eventually force my hand. v3: * PATCH 1+2: Update qapi-code-gen.rst [Kevin, Eric] * PATCH 4: Commit message typo [Eric], doc update moved to PATCH 2 * PATCH 5: Doc comment FIXME resolved [Kevin] v2: * Rebased with straightforward conflicts. * PATCH 1-4: No longer RFC. * PATCH 1: "Since" information fixed [Eric]. Commit message updated to reflect feedback. * PATCH 2: Commit message amended to point out special feature flag 'deprecated' is ignored at this stage. * PATCH 4: Documentation updated. Commit message tweaked. Markus Armbruster (5): qapi: Enable enum member introspection to show more than name qapi: Add feature flags to enum members qapi: Move compat policy from QObject to generic visitor qapi: Implement deprecated-input={reject,crash} for enum values block: Deprecate transaction type drive-backup docs/devel/qapi-code-gen.rst | 29 ++++++++++++++----- qapi/compat.json | 3 ++ qapi/introspect.json | 24 +++++++++++++-- qapi/transaction.json | 6 +++- include/qapi/qobject-input-visitor.h | 4 --- include/qapi/qobject-output-visitor.h | 4 --- include/qapi/util.h | 6 +++- include/qapi/visitor-impl.h | 3 ++ include/qapi/visitor.h | 9 ++++++ qapi/qapi-visit-core.c | 27 +++++++++++++++-- qapi/qmp-dispatch.c | 4 +-- qapi/qobject-input-visitor.c | 14 +-------- qapi/qobject-output-visitor.c | 14 +-------- scripts/qapi/expr.py | 3 +- scripts/qapi/introspect.py | 19 +++++++++--- scripts/qapi/schema.py | 22 ++++++++++++-- scripts/qapi/types.py | 17 ++++++++++- tests/qapi-schema/doc-good.json | 5 +++- tests/qapi-schema/doc-good.out | 3 ++ tests/qapi-schema/doc-good.txt | 3 ++ .../qapi-schema/enum-dict-member-unknown.err | 2 +- tests/qapi-schema/qapi-schema-test.json | 3 +- tests/qapi-schema/qapi-schema-test.out | 1 + tests/qapi-schema/test-qapi.py | 1 + 24 files changed, 164 insertions(+), 62 deletions(-) -- 2.31.1

1 6

Re: [Libguestfs] [libguestfs/nbdkit] nbdkit server hangs during negotiation when running in background, works in foreground (Issue #16)
by Richard W.M. Jones 24 Oct '21

24 Oct '21

On Sun, Oct 24, 2021 at 01:07:19AM -0700, Nathan Shearer wrote: > With my current setup, the issue is reproducible. If I start the > nbdkit server by placing a startup script in /etc/local.d to start > on boot, it will start, but the nbd-client will always hang during > negotiation. Killing and restarting the server and/or the client > does not resolve the deadlock. > > However, if I run the exact same script manually, by logging into > the server, then nbd-client is able to successfully connect to the > nbdkit server and everything works as expected. > > During my tests, I found that when I killed nbdkit, it would see > errors about their being insufficient "magic" or an invalid > password. I am not using any passwords, and both the nbdkit server > and nbd-client are on the same host, connecting through 127.0.0.1 I'm a bit unclear - how does it hang? What happens if you enable verbose messages (-v), what messages are sent to the log? Also try connecting with LIBNBD_DEBUG=1 nbdinfo nbd://server and see where it hangs in negotiation. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

1 1

libnbd | Failed pipeline for master | b456c9d6
by GitLab 23 Oct '21

23 Oct '21

Pipeline #394193393 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: b456c9d6 ( https://gitlab.com/nbdkit/libnbd/-/commit/b456c9d6a07e031031728b74e813a2317… ) Commit Message: golang/tests: Test expected Errno Use type ass... Commit Author: Nir Soffer Pipeline #394193393 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/394193393 ) triggered by Nir Soffer ( https://gitlab.com/nirs ) had 4 failed jobs. Job #1709015839 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1709015839/raw ) Stage: builds Name: x86_64-freebsd-13 Job #1709015838 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1709015838/raw ) Stage: builds Name: x86_64-freebsd-12 Job #1709015841 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1709015841/raw ) Stage: builds Name: x86_64-macos-11 Job #1709015840 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1709015840/raw ) Stage: builds Name: x86_64-freebsd-current -- You're receiving this email because of your account on gitlab.com.

1 0

[virt-v2v PATCH wave 1] lib/types: remove "source video: QXL" constructor
by Laszlo Ersek 21 Oct '21

21 Oct '21

* Background: We intend to stop virt-v2v from outputting converted domains with a QXL display device [1]. The reason is that QXL is a complex device model with a large security attack surface, while at the same time, better device models exist with regard to either guest compatibility, or graphics acceleration. Namely, if compatibility takes priority, standard VGA is at least as widely supported by guests [2] [3] [4], with lesser security risks. And if graphics performance takes priority, then 2D acceleration is generally considered obsolete, and virtio-vga is where new 3D features are being developed [5]. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1961107#c1 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1961107#c2 [3] https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/#qxl-vga [4] https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/#VGA [5] https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/#virtio-vga For virt-v2v in particular, compatibility is what matters, so standard VGA will ultimately supplant QXL in newly converted domains. * About this patch: In order to get the ball rolling, remove the "Source_QXL" constructor of the "source_video" union type. "Source_QXL" was originally introduced in commit [6], and then put to use in commit [7], specifically for the sake of in-place conversions, so that the "input" QXL display device serve as a distinguished hint for the "output" domain (after conversion). With the modularization of virt-v2v in commit [8], in-place conversion has been (temporarily) removed. Therefore, "Source_QXL" is functionally dead code, at the time of writing. [6] 04af2bc3cd2c ("v2v: collect source network and video adapter types", 2016-03-24) [7] 3c21ad036296 ("v2v: in-place: request caps based on source config", 2016-03-24) [8] 255722cbf39a ("v2v: Modular virt-v2v", 2021-09-07) "Functionally dead" means that, mentions of the QXL display device in the *source* domain descriptions of the test suite can be satisfied by the parametric "Source_other_video of string" constructor just as well; there is no actual functionality attached to "Source_QXL" at the moment. And once we reimplement in-place conversion for modular virt-v2v, we'll still want to flip domains that use QXL to standard VGA (see the background above). (Further patches are expected for RHBZ#1961107.) Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1961107 Signed-off-by: Laszlo Ersek <lersek(a)redhat.com> --- lib/types.ml | 4 +--- lib/types.mli | 2 +- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/types.ml b/lib/types.ml index 27f488255fd7..72d10d1ec774 100644 --- a/lib/types.ml +++ b/lib/types.ml @@ -90,7 +90,7 @@ and s_display_listen = | LNone and source_video = Source_other_video of string | - Source_Cirrus | Source_QXL + Source_Cirrus and source_sound = { s_sound_model : source_sound_model; @@ -260,12 +260,10 @@ and string_of_source_display { s_display_type = typ; ) and string_of_source_video = function - | Source_QXL -> "qxl" | Source_Cirrus -> "cirrus" | Source_other_video video -> video and source_video_of_string = function - | "qxl" -> Source_QXL | "cirrus" -> Source_Cirrus | video -> Source_other_video video diff --git a/lib/types.mli b/lib/types.mli index 2af5871fd54b..0bae445d8393 100644 --- a/lib/types.mli +++ b/lib/types.mli @@ -147,7 +147,7 @@ and s_display_listen = (** Video adapter model. *) and source_video = Source_other_video of string | - Source_Cirrus | Source_QXL + Source_Cirrus and source_sound = { s_sound_model : source_sound_model; (** Sound model. *) base-commit: 7ce5df273be304ec3ba87e46319cf78275eb4990 -- 2.19.1.3.g30247aa5d201

2 2

[PATCH v2 0/5] qapi: Add feature flags to enum members
by Markus Armbruster 21 Oct '21

21 Oct '21

PATCH 1+2 add feature flags to enum members. Awkward due to an introspection design mistake; see PATCH 1 for details. PATCH 3+4 implement policy deprecated-input={reject,crash} for enum values. Policy deprecated-output=hide is not implemented, because we can't hide a value without hiding the entire member, which is almost certainly more than the requester of this policy bargained for. Perhaps we want a new policy deprecated-output=hide-or-else-crash to help us catch unwanted use of deprecated enum values. Perhaps we want deprecated-output=hide to behave that way together with deprecated-input=crash. Or even always. Thoughts? PATCH 5 puts the new feature flags to use. It's RFC because it makes sense only on top of Vladimir's deprecation of drive-backup. See its commit message for a reference. I prefer to commit new features together with a use outside tests/. PATCH 5 adds such a use, but it's RFC, because it depends on Vladimir's work. Perhaps another use pops up. I can delay this work in the hope of a use becoming ready, but the feature flags work I have in the pipeline will eventually force my hand. v2: * Rebased with straightforward conflicts. * PATCH 1-4: No longer RFC. * PATCH 1: "Since" information fixed [Eric]. Commit message updated to reflect feedback. * PATCH 2: Commit message amended to point out special feature flag 'deprecated' is ignored at this stage. * PATCH 4: Documentation updated. Commit message tweaked. Markus Armbruster (5): qapi: Enable enum member introspection to show more than name qapi: Add feature flags to enum members qapi: Move compat policy from QObject to generic visitor qapi: Implement deprecated-input={reject,crash} for enum values block: Deprecate transaction type drive-backup docs/devel/qapi-code-gen.rst | 10 ++++--- qapi/compat.json | 3 +++ qapi/introspect.json | 24 +++++++++++++++-- qapi/transaction.json | 5 +++- include/qapi/qobject-input-visitor.h | 4 --- include/qapi/qobject-output-visitor.h | 4 --- include/qapi/util.h | 6 ++++- include/qapi/visitor-impl.h | 3 +++ include/qapi/visitor.h | 9 +++++++ qapi/qapi-visit-core.c | 27 ++++++++++++++++--- qapi/qmp-dispatch.c | 4 +-- qapi/qobject-input-visitor.c | 14 +--------- qapi/qobject-output-visitor.c | 14 +--------- scripts/qapi/expr.py | 3 ++- scripts/qapi/introspect.py | 19 ++++++++++--- scripts/qapi/schema.py | 22 +++++++++++++-- scripts/qapi/types.py | 17 +++++++++++- tests/qapi-schema/doc-good.json | 5 +++- tests/qapi-schema/doc-good.out | 3 +++ tests/qapi-schema/doc-good.txt | 3 +++ .../qapi-schema/enum-dict-member-unknown.err | 2 +- tests/qapi-schema/qapi-schema-test.json | 3 ++- tests/qapi-schema/qapi-schema-test.out | 1 + tests/qapi-schema/test-qapi.py | 1 + 24 files changed, 149 insertions(+), 57 deletions(-) -- 2.31.1

4 16

[PATCH nbdkit v2 0/2] New filter: nbdkit-retry-request-filter
by Richard W.M. Jones 19 Oct '21

19 Oct '21

I believe this addresses everything mentioned in the previous reviews (sorry if I missed anything). In addition I have extended web-server.c to implement a fairly realistic test scenario. Alexander: It would be useful if you could test this filter in your code as well so we can be sure it really fixes your problem. Rich.

4 11

[PATCH nbdkit] New filter: request-request
by Richard W.M. Jones 18 Oct '21

18 Oct '21

This is an incomplete and barely tested idea for a new filter that retries single requests, for: https://bugzilla.redhat.com/show_bug.cgi?id=2013000 Rich.

4 10

Rust update needed in stable branch?
by Richard W.M. Jones 16 Oct '21

16 Oct '21

[This time with the mailing list ...] Hi Alan, Quick question about: https://gitlab.com/nbdkit/nbdkit/-/commit/4818592227f61564ed2d54088ba3dc553… Is this something I should cherry pick for the stable branch (1.28.1)? How about for the old stable branch (1.26.x)? Or should I leave it for the development branch only for now? I'm not very clear if people use the old version of Rust or if everyone always uses the newest version. (Either way I'll cherry pick the repository URL fix from the patch) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

2 2

[PATCH nbdkit] curl: Add effective-url flag
by Richard W.M. Jones 14 Oct '21

14 Oct '21

Probably best to read this first: https://bugzilla.redhat.com/show_bug.cgi?id=2013000 This adds an effective-url=true|false flag to nbdkit-curl-plugin. If true, the first time we fetch the URL, we fetch the "effective" URL (ie. the URL after all redirects were resolved) and use that for all future connections within the current nbdkit instance. The idea behind this patch is to do with the Fedora mirror system which is flakey. Some mirrors return errors or 404s. And the mirror system will give you a new redirect each time (within your geographical region). This results in transfers sometimes failing because a single read went to a bad mirror. After implementing this patch I'm not very happy with it. It is incomplete because we pass the original URL to cookie-script and header-script, so plugin/curl/scripts.c will also need to be modified. Once those modifications are done the change becomes quite invasive. Also I'm not convinced that it really solves any problem. In the manual change I wrote: Note use of this feature in long-lived nbdkit instances can cause subtle problems: • The effective URL persists across connections for the lifetime of the nbdkit instance. If nbdkit is used for a long time then it is possible for the redirected URL to become stale. • It will defeat some mirror load-balancing techniques. • If the mirror service sometimes redirects to a broken URL and it happens that the URL you fetch first is broken then nbdkit will no longer recover on subsequent connections (instead you will need to restart nbdkit). I suggested another way to solve this by using curl APIs to fetch the effective URL up front and passing that URL to nbdkit (see https://bugzilla.redhat.com/show_bug.cgi?id=2013000#c1) but apparently that solution isn't acceptable for unclear reasons. I can't think of any other way to solve this in the context of nbdkit (maybe have it detect when redirection is happening and retry the redirection on error?). So here's the patch. Rich.

4 8