April 2023 - Libguestfs - libguestfs List Archives

[libnbd PATCH 00/23] continue wrapping generated C code harder
by Laszlo Ersek 02 May '23

02 May '23

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2172516 Repo: https://gitlab.com/lersek/libnbd.git Branch: wrap-2172516-wave4 This series is a continuation of the one noted in <https://bugzilla.redhat.com/show_bug.cgi?id=2172516#c15>. It reduces the widths of the following generated C files: file width before width after ------------------ ------------ ----------- include/libnbd.h 81 79 lib/api.c 206 79 ocaml/nbd-c.c 108 79 python/libnbdmod.c 130 79 python/methods.h 104 80 The following generated C files remain for the next wave: file width ------------------ ----- lib/states-run.c 102 lib/states.c 116 lib/states.h 123 Quite a few patches in the series are easier to review with "git show -b" or "git show --color-words", due to code reindentation or other relatively mechanical changes. (And I've just found out about "git show --color-moved", which works really well on patch #08, 'generator/utils: factor out "pr_buf"', for example.) Most patches where such display tweaks are beneficial mention that fact. These formatting tweaks are also the reason why I've pushed the branch to my libnbd fork on gitlab (see URL & branch name above). The posted patches should be possible to apply with git-am, for merging (thus, they couldn't be formatted with the "-b" option, for example), but for review, it may be easier to just fetch them (or to fetch them in addition) and to show them locally. Thanks for reviewing, Laszlo Laszlo Ersek (23): generator: split LIBNBD_ATTRIBUTE_NONNULL replacement text generator: wrap "python/methods.h" at 80 characters generator: wrap "python/libnbdmod.c" at 80 characters generator: split "int64_from_uint32_array" calls in "ocaml/nbd-c.c" generator: wrap nbd_internal_ocaml_raise_closed() calls in "ocaml/nbd-c.c" generator: factor out "num_args" in "print_ocaml_binding" generator: wrap byte-code compat function bodies in "ocaml/nbd-c.c" generator/utils: factor out "pr_buf" generator/utils: comment on "maxcol" handling in "pr_wrap" generator/utils: add "pr_wrap_cstr" generator/C: print_trace_enter: break "enter" to a common new line generator/C: print_trace_enter: expand "function" shorthand in value iters generator/C: print_trace_enter: print Closure values uniformly with others generator/C: print_trace_enter: generate commas as suffixes, not prefixes generator/C: print_trace_enter: strip spaces around PRI* macros generator/C: print_trace_enter: wrap "enter" debug() / debug_direct() call generator/C: print_wrapper: break apart the "ready" & "processing" states generator/C: print_wrapper: fold permitted states error message generator/C: print_trace_leave: separate arg list from function designator generator/C: print_trace_leave: wrap "leave" debug() / debug_direct() call generator/C: print_flags_check: break guard condition to a new line generator/C: print_wrapper: use helper variable for permitted state check generator/C: lib/api.c: indent arg list 2 spaces relative to function name generator/C.ml | 195 ++++++++++++-------- generator/OCaml.ml | 27 ++- generator/Python.ml | 13 +- generator/utils.ml | 85 ++++++++- generator/utils.mli | 1 + ocaml/helpers.c | 2 +- ocaml/nbd-c.h | 3 +- 7 files changed, 225 insertions(+), 101 deletions(-) base-commit: 0744f748ed900fb0537da9a5b6532538f3c78e83

3 29

[PATCH nbdkit 0/2] allocators: sparse: Split the highly contended mutex
by Richard W.M. Jones 02 May '23

02 May '23

nbdkit-memory-plugin is a RAM disk, but it is quite a slow one because of the lock we must acquire on the sparse array. Try to improve performance by changing this mutex for a read-write lock. The results are somewhat mixed. Note that in the real world, workloads which constantly update the metadata are assumed to be fairly rare. The test program added in the first patch exercises metadata updates a lot more aggressively than would be expected. Rich.

3 5

[PATCH v2] daemon/tar: support more compression methods.
by Zixun LI 02 May '23

02 May '23

Add support for lzma, zstd and auto detection of compression methods. Removed "auto" setting and updated documentation since v1. Signed-off-by: Zixun LI <admin(a)hifiphile.com> --- daemon/tar.c | 8 ++++++++ generator/actions_core.ml | 12 ++++++------ 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/daemon/tar.c b/daemon/tar.c index 57b5b37ff..ff29fe500 100644 --- a/daemon/tar.c +++ b/daemon/tar.c @@ -159,6 +159,10 @@ do_tar_in (const char *dir, const char *compress, int xattrs, int selinux, int a filter = " --xz"; else if (STREQ (compress, "lzop")) filter = " --lzop"; + else if (STREQ (compress, "lzma")) + filter = " --lzma"; + else if (STREQ (compress, "zstd")) + filter = " --zstd"; else { reply_with_error ("unknown compression type: %s", compress); return -1; @@ -312,6 +316,10 @@ do_tar_out (const char *dir, const char *compress, int numericowner, filter = " --xz"; else if (STREQ (compress, "lzop")) filter = " --lzop"; + else if (STREQ (compress, "lzma")) + filter = " --lzma"; + else if (STREQ (compress, "zstd")) + filter = " --zstd"; else { reply_with_error ("unknown compression type: %s", compress); return -1; diff --git a/generator/actions_core.ml b/generator/actions_core.ml index c8d9949ba..273ea1a7f 100644 --- a/generator/actions_core.ml +++ b/generator/actions_core.ml @@ -2710,9 +2710,9 @@ This command uploads and unpacks local file C<tarfile> into F<directory>. The optional C<compress> flag controls compression. If not given, then the input should be an uncompressed tar file. Otherwise one of the following strings may be given to select the compression -type of the input file: C<compress>, C<gzip>, C<bzip2>, C<xz>, C<lzop>. -(Note that not all builds of libguestfs will support all of these -compression types). +type of the input file: C<compress>, C<gzip>, C<bzip2>, C<xz>, C<lzop>, +C<lzma>, C<zstd>. (Note that not all builds of libguestfs will support +all of these compression types). The other optional arguments are: @@ -2745,9 +2745,9 @@ it to local file C<tarfile>. The optional C<compress> flag controls compression. If not given, then the output will be an uncompressed tar file. Otherwise one of the following strings may be given to select the compression -type of the output file: C<compress>, C<gzip>, C<bzip2>, C<xz>, C<lzop>. -(Note that not all builds of libguestfs will support all of these -compression types). +type of the output file: C<compress>, C<gzip>, C<bzip2>, C<xz>, C<lzop>, +C<lzma>, C<zstd>. (Note that not all builds of libguestfs will support +all of these compression types). The other optional arguments are: -- 2.40.1

2 1

[PATCH] daemon/tar: support more compression methods.
by Zixun LI 29 Apr '23

29 Apr '23

Add support for lzma, zstd and auto detection of compression methods. Signed-off-by: Zixun LI <admin(a)hifiphile.com> --- daemon/tar.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/daemon/tar.c b/daemon/tar.c index 57b5b37ff..347add341 100644 --- a/daemon/tar.c +++ b/daemon/tar.c @@ -159,6 +159,12 @@ do_tar_in (const char *dir, const char *compress, int xattrs, int selinux, int a filter = " --xz"; else if (STREQ (compress, "lzop")) filter = " --lzop"; + else if (STREQ (compress, "lzma")) + filter = " --lzma"; + else if (STREQ (compress, "zstd")) + filter = " --zstd"; + else if (STREQ (compress, "auto")) + filter = " --auto-compress"; else { reply_with_error ("unknown compression type: %s", compress); return -1; @@ -312,6 +318,12 @@ do_tar_out (const char *dir, const char *compress, int numericowner, filter = " --xz"; else if (STREQ (compress, "lzop")) filter = " --lzop"; + else if (STREQ (compress, "lzma")) + filter = " --lzma"; + else if (STREQ (compress, "zstd")) + filter = " --zstd"; + else if (STREQ (compress, "auto")) + filter = " --auto-compress"; else { reply_with_error ("unknown compression type: %s", compress); return -1; -- 2.40.1

2 1

Using nested partition with LUKS+GPT
by Philippe Midol-Monnet 29 Apr '23

29 Apr '23

Hi I have already fulfill a an issue, but it looks like it's not the recommended process. We are using a LUKS partition containing a GPT table and several partions. With libguestfs we can open the LUKS partition and the |/dev/mapper/<mapname>| device become available, but nested partition are not seen by the kernel. |partprobe| need to be run (or may be kpartx) to make partitions available in |/dev/mapper|. AFAIK there are no simple solution. As a workaround we use |guestfs_debug| but it's not a long term solution. We can see two solutions: * adding a udev rule that will automatically run partpobe when a new device appeared in /dev/mapper * adding a new API in libguestfs |guestfs_partprobe(device)| that can be call by the user when needed. What is the best best solution from your point of view? We can help implementing it. Regards Philippe Midol-Monnet -- *This email and any attachment contains EasyMile’s confidential information, and must not be modified or circulated without EasyMile’s prior written consent. It is intended exclusively for their recipient.s. If you received this message by mistake, please notify us promptly and immediately delete this email and any of its attachments.* *As data controller, EasyMile processes personal data, in compliance with the European GDPR (EU) 2016/679 of 27 April 2016. To exercise your rights, you can contact EasyMile at privacy(a)easymile.com, and if necessary, you may contact the local competent supervisory authority.”* *Cet e-mail et toute pièce jointe contiennent des informations confidentielles d'EasyMile et ne doivent pas être modifiés ou diffusés sans le consentement écrit préalable d'EasyMile. Ils sont destinés exclusivement à leur destinataire.s. Si vous avez reçu ce message par erreur, veuillez nous en informer rapidement et supprimer immédiatement cet e-mail et toutes ses pièces jointes. * *En tant que responsable de traitement, EasyMile traite des données personnelles, conformément au RGPD européen (UE) 2016/679 du 27 avril 2016. Pour exercer vos droits, vous pouvez contacter EasyMile à privacy(a)easymile.com <mailto:privacy@easymile.com>, et si nécessaire, vous pouvez contacter l'autorité de contrôle locale compétente.*

2 1

[PATCH 0/3] daemon: speed up SELinux relabeling in Linux guests
by Andrey Drobyshev 27 Apr '23

27 Apr '23

The first 2 patches are mere refactoring and small fixes. Initial discussion: https://listman.redhat.com/archives/libguestfs/2023-April/031367.html Andrey Drobyshev (3): daemon/selinux-relabel: don't exclude "/selinux" if it's non-existent daemon/selinux-relabel: search for "invalid option" in setfiles output daemon/selinux-relabel: run setfiles with "-T 0", if supported daemon/selinux-relabel.c | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) -- 2.31.1

3 9

[PATCH v3 0/6] NBD 64-bit extensions (spec only)
by Eric Blake 24 Apr '23

24 Apr '23

v2 was here: https://lists.debian.org/nbd/2022/11/msg00030.html The bulk of the changes since then are: - forbid NBD_OPT_EXPORT_NAME once extended headers are negotiated (Wouter) - consistently use 'maximum payload', rather than a haphazard mix of 'maximum block payload' (Vladimir) At this point, I want to make sure we are happy with the spec before re-posting patches for qemu and libnbd (there were some slight tweaks to the spec compared to v2, which will be easy enough to code, but I want to make sure there won't be any other tweaks needed based on review comments to this proposal). I could still be persuaded that if extended headers are negotiated, then the server should ONLY send back 64-bit block status replies (as written here, and as I implemented in v2 patches for qemu and libnbd, I specified that the server can choose either 32- or 64-bit replies, based on what fits best, and regardless of whether the client made a request with effect length less than 32 bits; clients must therefore be prepared to handle both types of replies). I'm still not sure whether patch 6 is desirable; it didn't get any feedback in my v2 posting. In relation to v2, the subject line for patch 2 changed (the 'git backport-diff' tool doesn't handle that well), and the interdiff changes look bigger than they really are due to reflowed paragraphs. [----] : patches are identical [####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively 001/6:[----] [-C] 'spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length' 002/6:[down] 'spec: Change maximum block size to maximum payload size' 003/6:[0237] [FC] 'spec: Add NBD_OPT_EXTENDED_HEADERS' 004/6:[----] [-C] 'spec: Allow 64-bit block status results' 005/6:[----] [-C] 'spec: Introduce NBD_FLAG_BLOCK_STATUS_PAYLOAD' 006/6:[0062] [FC] 'RFC: spec: Introduce NBD_REPLY_TYPE_OFFSET_HOLE_EXT' Eric Blake (6): spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length spec: Change maximum block size to maximum payload size spec: Add NBD_OPT_EXTENDED_HEADERS spec: Allow 64-bit block status results spec: Introduce NBD_FLAG_BLOCK_STATUS_PAYLOAD RFC: spec: Introduce NBD_REPLY_TYPE_OFFSET_HOLE_EXT doc/proto.md | 738 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 549 insertions(+), 189 deletions(-) base-commit: fd2c5a60e36ef0ea0d7fd6f8a6bd908d24d5801d -- 2.39.2

3 18

[PATCH] spec: Document extended headers extension branch
by Eric Blake 24 Apr '23

24 Apr '23

Branch extension-ext-header now exists to capture the specs reviewed on list and tentatively implemented in both qemu (server and client) and libnbd (client) code. I'm still working on implementing the extension in nbdkit (server) before promoting the extension branch to mainline. Meanwhile, structured replies and block status have been incorporated into mainline for some time now (see commit 6e896bca in 2018, v3.17); and while block status is still not implemented in the kernel module or in nbd-server, we did just recently implement structured replies in nbd-server. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- Given the review on my v3 spec proposal for extended headers, I've pushed patch 1 and 2 of that series to mainline along with this patch, while patches 3-5 of that series are tweaked per review comments and pushed on the new extension-ext-header branch. My next step will be posting v3 patches to qemu and libnbd which rebases my earlier proof-of-concept patches to those projects to match the spec as now live (and/or tweak the extension spec if something insurmountable comes up). doc/proto.md | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/doc/proto.md b/doc/proto.md index 460823b..8589e22 100644 --- a/doc/proto.md +++ b/doc/proto.md @@ -1137,6 +1137,9 @@ The field has the following format: will be faster than a regular write). Clients MUST NOT set the `NBD_CMD_FLAG_FAST_ZERO` request flag unless this transmission flag is set. +- bit 12, `NBD_FLAG_BLOCK_STATUS_PAYLOAD`; defined by the experimental + `EXTENDED_HEADERS` + [extension](https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-hea…. Clients SHOULD ignore unknown flags. @@ -1500,6 +1503,11 @@ of the newstyle negotiation. option does not select any metadata context, provided the client then does not attempt to issue `NBD_CMD_BLOCK_STATUS` commands. +* `NBD_OPT_EXTENDED_HEADERS` (11) + + Defined by the experimental `EXTENDED_HEADERS` + [extension](https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-hea…. + #### Option reply types These values are used in the "reply type" field, sent by the server @@ -1680,6 +1688,11 @@ case that data is an error message string suitable for display to the user. The request or the reply is too large to process. +* `NBD_REP_ERR_EXT_HEADER_REQD` (2^31 + 10) + + Defined by the experimental `EXTENDED_HEADERS` + [extension](https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-hea…. + ### Transmission phase #### Flag fields @@ -1733,6 +1746,9 @@ valid may depend on negotiation during the handshake phase. `NBD_CMD_WRITE`, then the server MUST fail quickly with an error of `NBD_ENOTSUP`. The client MUST NOT set this unless the server advertised `NBD_FLAG_SEND_FAST_ZERO`. +- bit 5, `NBD_CMD_FLAG_PAYLOAD_LEN`; defined by the experimental + `EXTENDED_HEADERS` + [extension](https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-hea…. ##### Structured reply flags @@ -1853,6 +1869,11 @@ small amount of fixed-length overhead inherent in the chunk type). extent information at the first offset not covered by a reduced-length reply. +* `NBD_REPLY_TYPE_BLOCK_STATUS_EXT` (6) + + Defined by the experimental `EXTENDED_HEADERS` + [extension](https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-hea…. + All error chunk types have bit 15 set, and begin with the same *error*, *message length*, and optional *message* fields as `NBD_REPLY_TYPE_ERROR`. If nonzero, *message length* indicates @@ -2336,9 +2357,7 @@ with names starting with the word 'extension'. Currently known are: -* The `STRUCTURED_REPLY` [extension](https://github.com/NetworkBlockDevice/nbd/blob/extension-structu…. - -* The `BLOCK_STATUS` [extension](https://github.com/NetworkBlockDevice/nbd/blob/extension-blockst… (based on the `STRUCTURED_REPLY` extension). +* The `EXTENDED_HEADER` [extension](https://github.com/NetworkBlockDevice/nbd/blob/extension-ext-hea…. * The `RESIZE` [extension](https://github.com/NetworkBlockDevice/nbd/blob/extension-resize/…. base-commit: daf3fce989b23cc8c9023f28e0eec2f074c9f1bd -- 2.40.0

1 0

Specify partition when using virt-customize --copy-in, copy file to boot partition
by Ingo Weinmann 24 Apr '23

24 Apr '23

Using virt-customize of libguestfs, I can copy a file from the host to the root partition of an image like this: virt-customize -a 2023-02-21-raspios-bullseye-arm64-lite.img --copy-in cmdline.txt:/boot/ But the image also has a boot partition (FAT) and I want to copy that file to that partition instead. Is that possible using virt-customize?

2 1

[COMMON PATCH 0/1] mlcustomize: skip SELinux relabeling if it's disabled
by Andrey Drobyshev 24 Apr '23

24 Apr '23

This patch effectively limits the number of cases when we would want to do a complete SELinux relabeling on Linux guest conversion. This was brought to my attention as we've recently had a support case when the conversion was taking too much time mostly because of relabeling performed with "setfiles -F". Although this patch might be worthy of taking as it is, I'd also like to clarify, do we really need relabeling of the entire file system during conversion? What exactly might go wrong here if we don't do that? Since this process might take hours on VMs big enough, it would be beneficial to be able to limit number of such cases even further, if possible. Unfortunately I couldn't find any hints in the libguestfs commit history as the relabeling code goes back to 0131d6f666 ("New tool: virt-v2v."). Roman Kagan (1): customize: skip SELinux relabeling if it's disabled mlcustomize/SELinux_relabel.ml | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) -- 2.31.1

4 9