February 2022 - Libguestfs - libguestfs List Archives

RFC for NBD protocol extension: extended headers
by Eric Blake 04 Oct '22

04 Oct '22

In response to this mail, I will be cross-posting a series of patches to multiple projects as a proof-of-concept implementation and request for comments on a new NBD protocol extension, called NBD_OPT_EXTENDED_HEADERS. With this in place, it will be possible for clients to request 64-bit zero, trim, cache, and block status operations when supported by the server. Not yet complete: an implementation of this in nbdkit. I also plan to tweak libnbd's 'nbdinfo --map' and 'nbdcopy' to take advantage of the larger block status results. Also, with 64-bit commands, we may want to also make it easier to let servers advertise an actual maximum size they are willing to accept for the commands in question (for example, a server may be happy with a full 64-bit block status, but still want to limit non-fast zero and cache to a smaller limit to avoid denial of service). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org

4 46

[PATCH libnbd 0/8] nbdcopy: Adaptive queue size
by Nir Soffer 06 Mar '22

06 Mar '22

This series add adaptive queue size feature, which gives great performance improvmemnt on my laptop, but less exciting results on a real server. When qemu-nbd will support MULTI-CON for writes, this should become more interesting. To implement this I added a worker struct for keeping worker state, and cleaned up the completion flow and other stuff. I think these cleanups are a good idea even if we do not add adaptive queue size. Nir Soffer (8): copy: Remove wrong references to holes copy: Rename copy_subcommand to create_subcommand copy: Extract create_command and create_buffer helpers copy: Separate finishing a command from freeing it copy: Introduce worker struct copy: Keep worker pointer in command copy: Track worker queue size copy: Adaptive queue size copy/file-ops.c | 4 +- copy/main.c | 58 +++++--- copy/multi-thread-copying.c | 270 ++++++++++++++++++++++-------------- copy/nbd-ops.c | 16 +-- copy/nbdcopy.h | 31 +++-- copy/nbdcopy.pod | 12 +- copy/null-ops.c | 4 +- copy/pipe-ops.c | 2 +- 8 files changed, 248 insertions(+), 149 deletions(-) -- 2.35.1

3 30

[v2v PATCH] m4: require libguestfs >= 1.44
by Laszlo Ersek 01 Mar '22

01 Mar '22

The next libguestfs-common submodule update will assume GUESTFS_HAVE_LUKS_UUID and GUESTFS_HAVE_CRYPTSETUP_OPEN; guestfs_cryptsetup_open() is available starting with libguestfs-1.44. Suggested-by: Richard W.M. Jones <rjones(a)redhat.com> Signed-off-by: Laszlo Ersek <lersek(a)redhat.com> --- m4/guestfs-libraries.m4 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/m4/guestfs-libraries.m4 b/m4/guestfs-libraries.m4 index 3994eeb26eda..551b2d82637b 100644 --- a/m4/guestfs-libraries.m4 +++ b/m4/guestfs-libraries.m4 @@ -18,7 +18,7 @@ dnl Any C libraries required by virt-v2v. dnl Of course we need libguestfs. -PKG_CHECK_MODULES([LIBGUESTFS], [libguestfs >= 1.40]) +PKG_CHECK_MODULES([LIBGUESTFS], [libguestfs >= 1.44]) dnl And libnbd. PKG_CHECK_MODULES([LIBNBD], [libnbd]) -- 2.19.1.3.g30247aa5d201

2 2

[libguestfs-common PATCH 0/3] cleanups for "options/decrypt.c"
by Laszlo Ersek 01 Mar '22

01 Mar '22

Three patches to address the comments that Rich made (on preexistent code) in this thread: [libguestfs-common PATCH 0/2] options: decrypt LUKS-on-LV devices Message-Id: <20220223161915.16604-1-lersek(a)redhat.com> https://listman.redhat.com/archives/libguestfs/2022-February/msg00369.html Thanks, Laszlo Laszlo Ersek (3): options: assume GUESTFS_HAVE_LUKS_UUID and GUESTFS_HAVE_CRYPTSETUP_OPEN options: remove stale comment about GUESTFS_CRYPTSETUP_OPEN_READONLY options: allocate the decrypted LUKS device name dynamically options/decrypt.c | 90 ++++++++++++++------ 1 file changed, 63 insertions(+), 27 deletions(-) base-commit: 2d8c0f8d40b5d43e45b969e25eb5c4a12b3c675a -- 2.19.1.3.g30247aa5d201

1 4

add some support for LUKS-on-LVM
by Laszlo Ersek 28 Feb '22

28 Feb '22

Hi, for <https://bugzilla.redhat.com/show_bug.cgi?id=1658126>, I'll soon post four short patch sets in response to this email (I'm sending this email basically as a common cover letter for those sets, for better threading). The main difference (for libguestfs & co) between LUKS-on-LVM and LVM-on-LUKS is that in the latter, the names of the decrypted LUKS block devices don't matter. However we name the decrypted devices via inspect_do_decrypt() -> make_mapname() in "common/options/decrypt.c" is irrelevant, as LVM will scan for PVs on all block devices, regardless of their names. The resultant LVs (hosting the filesystems) are what "/etc/fstab" refers to, and we can deal with those names -- the VG and LV names -- well, because (a) there is an "lvs" command (and API) for enumeration (see "lv_canonical" in "daemon/lvm_utils.ml", and lv_canonical() in "daemon/lvm.c" -- see also commit c37c8e770d664), and (b) the VG and LV names are stored in the LVM metadata, so libguestfs doesn't have to "invent" names. This is not the case with LUKS-on-LVM. The names of the decrypted LUKS devices is relevant, as they directly host filesystems, and "/etc/fstab" likely refers to them. Furthermore, there is no enumeration / scanning inside the guest, only a file called "/etc/crypttab". Unfortunately, the root filesystem containing "/etc/crypttab" tends to exist on a LUKS device as well, so we can't consult it for determining the decrypted device names when we're about to decrypt those same devices in inspect_do_decrypt(). (This circular dependency is circumvented during normal guest boot by having a plaintext initrd containing a copy of "/etc/crypttab", and LUKS-related parameters on the a kernel command line in the bootloader config.) In fact, since at least RHEL6, when the user selects either LVM-on-LUKS or LUKS-on-LVM in Anaconda, the installer never asks for LUKS device names (while it does let the user set VG and LV names). Instead, for a given LUKS header containing UUID <UUID>, the decrypted block device will automatically be named /dev/mapper/luks-<UUID>. (Anaconda sets up "/etc/crypttab" and "/etc/fstab" like that.) The patches I'm about to post follow the same logic -- I don't attempt intricate UUID-based (re)mappings through "/etc/crypttab", in "check_fstab_entry" -> "resolve_fstab_device", in "daemon/inspect_fs_unix_fstab.ml". (Assume that "/etc/fstab" refers to "/dev/mapper/hunter2" as a plaintext block device to mount. Assume that "/etc/crypttab" associates the "hunter2" device name with a block device containing a LUKS header with UUID 28372dec-afca-412a-a089-736437fd03af. (For us to realize this, we'd need to parse "/etc/crypttab" with Augeas in the first place.) Then we'd have to recall, by UUID, that we had mapped that particular LUKS device as "/dev/mapper/crypt<WHATEVER>" in inspect_do_decrypt(). Then either close and reopen the LUKS device as "hunter2" (there is no rename operation!), which could get messy as the dependent filesystem would be in use, or replace "hunter2" with "crypt<WHATEVER>" in both "/etc/fstab" and "/etc/crypttab".) Instead, accept the "/dev/mapper/luks-<UUID>" naming scheme as the standard one. Thus, inspect_do_decrypt() will name the decrypted block device "/dev/mapper/luks-<UUID>" at once (rather than "/dev/mapper/crypt<WHATEVER>"). And that name is expected to match the references in "/etc/fstab" and "/etc/crypttab". I've tested this with RHEL-6 and RHEL-7 guests successfully (using "guestfish -i", "virt-inspector", and virt-v2v). Example "lsblk" outputs, with "lsblk" executed in the converted domains: RHEL6: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom vda 252:0 0 20G 0 disk ├─vda1 252:1 0 500M 0 part /boot └─vda2 252:2 0 19.5G 0 part ├─vg_rhel6luks-root (dm-0) 253:0 0 14.7G 0 lvm │ └─luks-0d619854-ccd5-43b1-8883-991fec5ef713 (dm-2) 253:2 0 14.7G 0 crypt / └─vg_rhel6luks-swap (dm-1) 253:1 0 4.9G 0 lvm └─luks-4e9e7a6f-a68c-42fd-92b4-8f4f2579a389 (dm-3) 253:3 0 4.9G 0 crypt [SWAP] RHEL7: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 20G 0 disk ├─vda1 252:1 0 1G 0 part /boot └─vda2 252:2 0 19G 0 part ├─rhel_rhel7luks-root 253:0 0 10G 0 lvm │ └─luks-74558d73-1ef3-4327-878b-f11b266a20fc 253:3 0 10G 0 crypt / ├─rhel_rhel7luks-swap 253:1 0 3.9G 0 lvm │ └─luks-621270fd-6758-4001-8335-492cf5e32a45 253:2 0 3.9G 0 crypt [SWAP] └─rhel_rhel7luks-home 253:4 0 5.1G 0 lvm └─luks-0433f315-afd7-41a0-bd94-3f742114035c 253:5 0 5.1G 0 crypt /home Thanks, Laszlo

2 26

libnbd | Failed pipeline for master | 1402be98
by GitLab 24 Feb '22

24 Feb '22

Pipeline #479004232 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: 1402be98 ( https://gitlab.com/nbdkit/libnbd/-/commit/1402be98fc4a4d5493cc5eeb479ad3f91… ) Commit Message: docs: Document block size constraints in the ge... Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #479004232 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/479004232 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 1 failed job. Job #2134895162 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134895162/raw ) Stage: builds Name: x86_64-debian-10 -- You're receiving this email because of your account on gitlab.com.

1 0

libnbd | Failed pipeline for master | 5c0b2a57
by GitLab 24 Feb '22

24 Feb '22

Pipeline #478997323 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: 5c0b2a57 ( https://gitlab.com/nbdkit/libnbd/-/commit/5c0b2a576165ee363f5dcf70011fde0d3… ) Commit Message: interop: More test improvements The x86_64-ubu... Commit Author: Eric Blake ( https://gitlab.com/ebblake ) Pipeline #478997323 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/478997323 ) triggered by Eric Blake ( https://gitlab.com/ebblake ) had 1 failed job. Job #2134854248 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134854248/raw ) Stage: builds Name: x86_64-debian-10 -- You're receiving this email because of your account on gitlab.com.

1 0

libnbd | Failed pipeline for master | acc1f7a3
by GitLab 24 Feb '22

24 Feb '22

Pipeline #478984138 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: acc1f7a3 ( https://gitlab.com/nbdkit/libnbd/-/commit/acc1f7a389a868bf084011fcf67315123… ) Commit Message: docs/libnbd-release-notes-1.12.pod: Add credit ... Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #478984138 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/478984138 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 3 failed jobs. Job #2134776384 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134776384/raw ) Stage: builds Name: x86_64-debian-10 Job #2134776399 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134776399/raw ) Stage: builds Name: x86_64-opensuse-tumbleweed Job #2134776401 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134776401/raw ) Stage: builds Name: x86_64-ubuntu-1804 -- You're receiving this email because of your account on gitlab.com.

1 0

libnbd | Failed pipeline for master | baae33e2
by GitLab 24 Feb '22

24 Feb '22

Pipeline #478964498 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: baae33e2 ( https://gitlab.com/nbdkit/libnbd/-/commit/baae33e2768fbe5c19fc09637b212b6e0… ) Commit Message: interop: Skip block-size tests on old qemu-nbd ... Commit Author: Eric Blake ( https://gitlab.com/ebblake ) Pipeline #478964498 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/478964498 ) triggered by Eric Blake ( https://gitlab.com/ebblake ) had 4 failed jobs. Job #2134667068 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134667068/raw ) Stage: builds Name: x86_64-ubuntu-1804 Job #2134667072 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134667072/raw ) Stage: builds Name: x86_64-freebsd-12 Job #2134667059 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134667059/raw ) Stage: builds Name: x86_64-debian-10 Job #2134667067 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134667067/raw ) Stage: builds Name: x86_64-opensuse-tumbleweed -- You're receiving this email because of your account on gitlab.com.

1 0

libnbd | Failed pipeline for master | 270d3f0e
by GitLab 24 Feb '22

24 Feb '22

Pipeline #478882786 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: 270d3f0e ( https://gitlab.com/nbdkit/libnbd/-/commit/270d3f0e177449ee6edfab71e9560db37… ) Commit Message: Version 1.12.0. Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #478882786 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/478882786 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 3 failed jobs. Job #2134239719 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134239719/raw ) Stage: builds Name: x86_64-ubuntu-1804 Job #2134239707 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134239707/raw ) Stage: builds Name: x86_64-debian-10 Job #2134239718 ( https://gitlab.com/nbdkit/libnbd/-/jobs/2134239718/raw ) Stage: builds Name: x86_64-opensuse-tumbleweed -- You're receiving this email because of your account on gitlab.com.

1 0