RFC for NBD protocol extension: extended headers
by Eric Blake
In response to this mail, I will be cross-posting a series of patches
to multiple projects as a proof-of-concept implementation and request
for comments on a new NBD protocol extension, called
NBD_OPT_EXTENDED_HEADERS. With this in place, it will be possible for
clients to request 64-bit zero, trim, cache, and block status
operations when supported by the server.
Not yet complete: an implementation of this in nbdkit. I also plan to
tweak libnbd's 'nbdinfo --map' and 'nbdcopy' to take advantage of the
larger block status results. Also, with 64-bit commands, we may want
to also make it easier to let servers advertise an actual maximum size
they are willing to accept for the commands in question (for example,
a server may be happy with a full 64-bit block status, but still want
to limit non-fast zero and cache to a smaller limit to avoid denial of
service).
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
2 years, 2 months
[PATCH libnbd 0/8] nbdcopy: Adaptive queue size
by Nir Soffer
This series add adaptive queue size feature, which gives great
performance improvmemnt on my laptop, but less exciting results on a
real server. When qemu-nbd will support MULTI-CON for writes, this
should become more interesting.
To implement this I added a worker struct for keeping worker state, and
cleaned up the completion flow and other stuff. I think these cleanups
are a good idea even if we do not add adaptive queue size.
Nir Soffer (8):
copy: Remove wrong references to holes
copy: Rename copy_subcommand to create_subcommand
copy: Extract create_command and create_buffer helpers
copy: Separate finishing a command from freeing it
copy: Introduce worker struct
copy: Keep worker pointer in command
copy: Track worker queue size
copy: Adaptive queue size
copy/file-ops.c | 4 +-
copy/main.c | 58 +++++---
copy/multi-thread-copying.c | 270 ++++++++++++++++++++++--------------
copy/nbd-ops.c | 16 +--
copy/nbdcopy.h | 31 +++--
copy/nbdcopy.pod | 12 +-
copy/null-ops.c | 4 +-
copy/pipe-ops.c | 2 +-
8 files changed, 248 insertions(+), 149 deletions(-)
--
2.35.1
2 years, 9 months
[v2v PATCH] m4: require libguestfs >= 1.44
by Laszlo Ersek
The next libguestfs-common submodule update will assume
GUESTFS_HAVE_LUKS_UUID and GUESTFS_HAVE_CRYPTSETUP_OPEN;
guestfs_cryptsetup_open() is available starting with libguestfs-1.44.
Suggested-by: Richard W.M. Jones <rjones(a)redhat.com>
Signed-off-by: Laszlo Ersek <lersek(a)redhat.com>
---
m4/guestfs-libraries.m4 | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/m4/guestfs-libraries.m4 b/m4/guestfs-libraries.m4
index 3994eeb26eda..551b2d82637b 100644
--- a/m4/guestfs-libraries.m4
+++ b/m4/guestfs-libraries.m4
@@ -18,7 +18,7 @@
dnl Any C libraries required by virt-v2v.
dnl Of course we need libguestfs.
-PKG_CHECK_MODULES([LIBGUESTFS], [libguestfs >= 1.40])
+PKG_CHECK_MODULES([LIBGUESTFS], [libguestfs >= 1.44])
dnl And libnbd.
PKG_CHECK_MODULES([LIBNBD], [libnbd])
--
2.19.1.3.g30247aa5d201
2 years, 9 months
[libguestfs-common PATCH 0/3] cleanups for "options/decrypt.c"
by Laszlo Ersek
Three patches to address the comments that Rich made (on preexistent
code) in this thread:
[libguestfs-common PATCH 0/2] options: decrypt LUKS-on-LV devices
Message-Id: <20220223161915.16604-1-lersek(a)redhat.com>
https://listman.redhat.com/archives/libguestfs/2022-February/msg00369.html
Thanks,
Laszlo
Laszlo Ersek (3):
options: assume GUESTFS_HAVE_LUKS_UUID and
GUESTFS_HAVE_CRYPTSETUP_OPEN
options: remove stale comment about GUESTFS_CRYPTSETUP_OPEN_READONLY
options: allocate the decrypted LUKS device name dynamically
options/decrypt.c | 90 ++++++++++++++------
1 file changed, 63 insertions(+), 27 deletions(-)
base-commit: 2d8c0f8d40b5d43e45b969e25eb5c4a12b3c675a
--
2.19.1.3.g30247aa5d201
2 years, 9 months
add some support for LUKS-on-LVM
by Laszlo Ersek
Hi,
for <https://bugzilla.redhat.com/show_bug.cgi?id=1658126>, I'll soon
post four short patch sets in response to this email (I'm sending this
email basically as a common cover letter for those sets, for better
threading).
The main difference (for libguestfs & co) between LUKS-on-LVM and
LVM-on-LUKS is that in the latter, the names of the decrypted LUKS block
devices don't matter. However we name the decrypted devices via
inspect_do_decrypt() -> make_mapname() in "common/options/decrypt.c" is
irrelevant, as LVM will scan for PVs on all block devices, regardless of
their names. The resultant LVs (hosting the filesystems) are what
"/etc/fstab" refers to, and we can deal with those names -- the VG and
LV names -- well, because (a) there is an "lvs" command (and API) for
enumeration (see "lv_canonical" in "daemon/lvm_utils.ml", and
lv_canonical() in "daemon/lvm.c" -- see also commit c37c8e770d664), and
(b) the VG and LV names are stored in the LVM metadata, so libguestfs
doesn't have to "invent" names.
This is not the case with LUKS-on-LVM. The names of the decrypted LUKS
devices is relevant, as they directly host filesystems, and "/etc/fstab"
likely refers to them. Furthermore, there is no enumeration / scanning
inside the guest, only a file called "/etc/crypttab". Unfortunately, the
root filesystem containing "/etc/crypttab" tends to exist on a LUKS
device as well, so we can't consult it for determining the decrypted
device names when we're about to decrypt those same devices in
inspect_do_decrypt(). (This circular dependency is circumvented during
normal guest boot by having a plaintext initrd containing a copy of
"/etc/crypttab", and LUKS-related parameters on the a kernel command
line in the bootloader config.)
In fact, since at least RHEL6, when the user selects either LVM-on-LUKS
or LUKS-on-LVM in Anaconda, the installer never asks for LUKS device
names (while it does let the user set VG and LV names). Instead, for a
given LUKS header containing UUID <UUID>, the decrypted block device
will automatically be named /dev/mapper/luks-<UUID>. (Anaconda sets up
"/etc/crypttab" and "/etc/fstab" like that.)
The patches I'm about to post follow the same logic -- I don't attempt
intricate UUID-based (re)mappings through "/etc/crypttab", in
"check_fstab_entry" -> "resolve_fstab_device", in
"daemon/inspect_fs_unix_fstab.ml".
(Assume that "/etc/fstab" refers to "/dev/mapper/hunter2" as a plaintext
block device to mount.
Assume that "/etc/crypttab" associates the "hunter2" device name with a
block device containing a LUKS header with UUID
28372dec-afca-412a-a089-736437fd03af. (For us to realize this, we'd need
to parse "/etc/crypttab" with Augeas in the first place.)
Then we'd have to recall, by UUID, that we had mapped that particular
LUKS device as "/dev/mapper/crypt<WHATEVER>" in inspect_do_decrypt().
Then either close and reopen the LUKS device as "hunter2" (there is no
rename operation!), which could get messy as the dependent filesystem
would be in use, or replace "hunter2" with "crypt<WHATEVER>" in both
"/etc/fstab" and "/etc/crypttab".)
Instead, accept the "/dev/mapper/luks-<UUID>" naming scheme as the
standard one. Thus, inspect_do_decrypt() will name the decrypted block
device "/dev/mapper/luks-<UUID>" at once (rather than
"/dev/mapper/crypt<WHATEVER>"). And that name is expected to match the
references in "/etc/fstab" and "/etc/crypttab".
I've tested this with RHEL-6 and RHEL-7 guests successfully (using
"guestfish -i", "virt-inspector", and virt-v2v). Example "lsblk"
outputs, with "lsblk" executed in the converted domains:
RHEL6:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 500M 0 part /boot
└─vda2 252:2 0 19.5G 0 part
├─vg_rhel6luks-root (dm-0) 253:0 0 14.7G 0 lvm
│ └─luks-0d619854-ccd5-43b1-8883-991fec5ef713 (dm-2)
253:2 0 14.7G 0 crypt /
└─vg_rhel6luks-swap (dm-1) 253:1 0 4.9G 0 lvm
└─luks-4e9e7a6f-a68c-42fd-92b4-8f4f2579a389 (dm-3)
253:3 0 4.9G 0 crypt [SWAP]
RHEL7:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 19G 0 part
├─rhel_rhel7luks-root 253:0 0 10G 0 lvm
│ └─luks-74558d73-1ef3-4327-878b-f11b266a20fc 253:3 0 10G 0 crypt /
├─rhel_rhel7luks-swap 253:1 0 3.9G 0 lvm
│ └─luks-621270fd-6758-4001-8335-492cf5e32a45 253:2 0 3.9G 0 crypt [SWAP]
└─rhel_rhel7luks-home 253:4 0 5.1G 0 lvm
└─luks-0433f315-afd7-41a0-bd94-3f742114035c 253:5 0 5.1G 0 crypt /home
Thanks,
Laszlo
2 years, 10 months
