March 2022 - Libguestfs - libguestfs List Archives

RFC for NBD protocol extension: extended headers
by Eric Blake 04 Oct '22

04 Oct '22

In response to this mail, I will be cross-posting a series of patches to multiple projects as a proof-of-concept implementation and request for comments on a new NBD protocol extension, called NBD_OPT_EXTENDED_HEADERS. With this in place, it will be possible for clients to request 64-bit zero, trim, cache, and block status operations when supported by the server. Not yet complete: an implementation of this in nbdkit. I also plan to tweak libnbd's 'nbdinfo --map' and 'nbdcopy' to take advantage of the larger block status results. Also, with 64-bit commands, we may want to also make it easier to let servers advertise an actual maximum size they are willing to accept for the commands in question (for example, a server may be happy with a full 64-bit block status, but still want to limit non-fast zero and cache to a smaller limit to avoid denial of service). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org

4 46

[p2v PATCH 00/10] Simplify NBD server management
by Laszlo Ersek 01 Apr '22

01 Apr '22

Repo: https://github.com/lersek/virt-p2v.git Branch: nbd-servers-simplification Rich proposed a list of tasks at <https://listman.redhat.com/archives/libguestfs/2022-March/028475.html>, for simplifying NBD server management. This series implements them. Earlier I said I'd post this as an RFC series, but after reviewing and testing the set myself, I decided to post it as a PATCH series. The set builds at every stage. I verified the documentation ("docs/p2v-building.1" and "virt-p2v.1") at every stage that modifies the sources of those files, too. The set passes make check I also tested it with make run-virt-p2v-in-an-nvme-vm i.e. an actual conversion, running against a RHEL9 conversion server (virt-v2v 1.45.99rhel=9,release=1.el9; nbdkit 1.30.1-1.el9.x86_64). The converted guest boots fine. The conversion log is available if needed. On the p2v (virtual) machine, I saved the interesting messages from the serial console (they are interesting because they relate to code modified by this series): > [ 232.205162] launch-virt-p2v[669]: checking for nbdkit ... > [ 233.209388] launch-virt-p2v[675]: nbdkit 1.28.6 (nbdkit-1.28.6-1.fc35) > [ 233.213329] launch-virt-p2v[675]: file 1.28.6 > [ 233.216906] launch-virt-p2v[669]: found nbdkit > > [ 301.197612] launch-virt-p2v[669]: virt-p2v: bound to localhost:50123 (2 socket(s)) > [ 301.198566] launch-virt-p2v[669]: starting nbdkit for /dev/nvme0n1 using socket activation > [ 301.220919] launch-virt-p2v[669]: ssh command: ssh -p 22 -l root -o StrictHostKeyChecking=no -o ConnectTimeout=60 -o ServerAliveInterval=300 -o ServerAliveCountMax=6 -o PreferredAuthentications=keyboard-interactive,password -R 0:localhost:50123 -N 192.168.122.184 > [ 301.225792] launch-virt-p2v[748]: nbdkit: file[1]: error: reading initial client flags: conn->recv: Connection reset by peer > > [ 301.583073] launch-virt-p2v[669]: virt-p2v: data connection for /dev/nvme0n1: SSH remote port 44479, local port 50123 > > [ 301.588509] launch-virt-p2v[669]: virt-p2v: bound to localhost:50124 (2 socket(s)) > [ 301.592034] launch-virt-p2v[669]: starting nbdkit for /dev/nvme1n1 using socket activation > [ 301.605749] launch-virt-p2v[669]: ssh command: ssh -p 22 -l root -o StrictHostKeyChecking=no -o ConnectTimeout=60 -o ServerAliveInterval=300 -o ServerAliveCountMax=6 -o PreferredAuthentications=keyboard-interactive,password -R 0:localhost:50124 -N 192.168.122.184 > [ 301.612459] launch-virt-p2v[751]: nbdkit: file[1]: error: reading initial client flags: conn->recv: Connection reset by peer > > [ 302.002998] launch-virt-p2v[669]: virt-p2v: data connection for /dev/nvme1n1: SSH remote port 35789, local port 50124 > > [ 302.007171] launch-virt-p2v[669]: virt-p2v: bound to localhost:50125 (2 socket(s)) > [ 302.009976] launch-virt-p2v[669]: starting nbdkit for /dev/nvme1n2 using socket activation > [ 302.020637] launch-virt-p2v[669]: ssh command: ssh -p 22 -l root -o StrictHostKeyChecking=no -o ConnectTimeout=60 -o ServerAliveInterval=300 -o ServerAliveCountMax=6 -o PreferredAuthentications=keyboard-interactive,password -R 0:localhost:50125 -N 192.168.122.184 > [ 302.026882] launch-virt-p2v[754]: nbdkit: file[1]: error: reading initial client flags: conn->recv: Connection reset by peer > > [ 302.410966] launch-virt-p2v[669]: virt-p2v: data connection for /dev/nvme1n2: SSH remote port 38663, local port 50125 I formatted the series with "-W" (aka "--function-context"), for easier review. Some patches are best viewed with "-b" (aka "--ignore-space-change"), for masking un-indentation; I did not use that option for posting this series, as it could interfere with applying the patches from the list. For looking at some of the patches like that, fetch the series from the URL at the top (or apply it from the list), and run "git show -b" locally. Thanks, Laszlo Laszlo Ersek (10): remove qemu-nbd support remove "--nbd=nbdkit-no-sa" option parsing nbd.c: simplify start_nbdkit() open-code "localhost" as the listen hostname for nbdkit nbd.c: remove bind_source_port() remove "--nbd" option parsing nbd.c: remove nbd_server_string() nbd.c: remove "enum nbd_server" rely on Linux for killing nbdkit, when the parent thread exits nbd.c: bind listening socket without AI_ADDRCONFIG Makefile.am | 2 - contrib/aux-scripts/do-build.sh | 7 +- contrib/patches/0002-RHEL-5-ONLY-QEMU-NBD-1.4-HAS-NO-f-OPTION.patch | 34 -- conversion.c | 23 +- dependencies.m4 | 6 +- docs/p2v-building.pod | 9 +- main.c | 7 +- nbd.c | 514 +++----------------- p2v.h | 10 +- ssh.c | 6 +- test-virt-p2v-nbdkit.sh | 6 +- test-virt-p2v-scp.sh | 2 +- test-virt-p2v-ssh.sh | 4 +- test-virt-p2v.sh | 56 --- virt-p2v.pod | 49 +- 15 files changed, 109 insertions(+), 626 deletions(-) delete mode 100644 contrib/patches/0002-RHEL-5-ONLY-QEMU-NBD-1.4-HAS-NO-f-OPTION.patch delete mode 100755 test-virt-p2v.sh base-commit: c1a4bd83a8bba5959470ef1f6deb06dc61587112 -- 2.19.1.3.g30247aa5d201

2 27

[v2v PATCH] convert/linux_kernels: fix virtio-socket check
by Laszlo Ersek 01 Apr '22

01 Apr '22

The virtio-vsock support check from commit 05f780c16f01 ("v2v: support configuration of viosock driver", 2021-02-26) is wrong; it looks for one of two things: - a module called "virtio_socket", - a kernel config item called VIRTIO_SOCKET. In reality, we need: - the "vmw_vsock_virtio_transport" module, - or the kernel config item VIRTIO_VSOCKETS. Refer to Linux commit 304ba62fd4e6 ("VSOCK: Add Makefile and Kconfig", 2016-08-02). Fix the parameters of the related "kernel_supports" call. The same virt-v2v commit also missed extending the "print_kernel_info" function; do that now as well. I tested this patch by converting a RHEL-8.5 guest from libvirt to libvirt (originally creating the guest with virt-builder). Relevant differences between the "before" and "after" conversion logs: > installed kernel packages in this guest: > * kernel-core 4.18.0-348.el8.x86_64 (x86_64) > /boot/vmlinuz-4.18.0-348.el8.x86_64 > /boot/initramfs-4.18.0-348.el8.x86_64.img > /boot/config-4.18.0-348.el8.x86_64 > /lib/modules/4.18.0-348.el8.x86_64 > 2580 modules found > virtio: blk=true net=true rng=true balloon=true > - pvpanic=true xen=false debug=false > + pvpanic=true vsock=true xen=false debug=false and > kernels offered by the bootloader in this guest (first in list is default): > * kernel-core 4.18.0-348.el8.x86_64 (x86_64) > /boot/vmlinuz-4.18.0-348.el8.x86_64 > /boot/initramfs-4.18.0-348.el8.x86_64.img > /boot/config-4.18.0-348.el8.x86_64 > /lib/modules/4.18.0-348.el8.x86_64 > 2580 modules found > virtio: blk=true net=true rng=true balloon=true > - pvpanic=true xen=false debug=false > + pvpanic=true vsock=true xen=false debug=false > * kernel-core 4.18.0-348.el8.x86_64 (x86_64) > /boot/vmlinuz-4.18.0-348.el8.x86_64 > /boot/initramfs-4.18.0-348.el8.x86_64.img > /boot/config-4.18.0-348.el8.x86_64 > /lib/modules/4.18.0-348.el8.x86_64 > 2580 modules found > virtio: blk=true net=true rng=true balloon=true > - pvpanic=true xen=false debug=false > + pvpanic=true vsock=true xen=false debug=false and > best kernel for this guest: > * kernel-core 4.18.0-348.el8.x86_64 (x86_64) > /boot/vmlinuz-4.18.0-348.el8.x86_64 > /boot/initramfs-4.18.0-348.el8.x86_64.img > /boot/config-4.18.0-348.el8.x86_64 > /lib/modules/4.18.0-348.el8.x86_64 > 2580 modules found > virtio: blk=true net=true rng=true balloon=true > - pvpanic=true xen=false debug=false > + pvpanic=true vsock=true xen=false debug=false and > gcaps_block_bus = virtio-blk > gcaps_net_bus = virtio-net > gcaps_virtio_rng = true > gcaps_virtio_balloon = true > gcaps_isa_pvpanic = true > -gcaps_virtio_socket = false > +gcaps_virtio_socket = true > gcaps_machine = q35 > gcaps_arch = x86_64 > gcaps_acpi = true > gcaps_virtio_1_0 = true and > resulting XML for libvirt: > <?xml version='1.0' encoding='utf-8'?> > <domain type='kvm'> >  > <name>converted</name> > <metadata> > <libosinfo:libosinfo > xmlns:libosinfo='http://libosinfo.org/xmlns/libvirt/domain/1.0'> > <libosinfo:os id='http://redhat.com/rhel/8.5'/> > </libosinfo:libosinfo> > </metadata> > <memory unit='KiB'>2097152</memory> > <currentMemory unit='KiB'>2097152</currentMemory> > <vcpu>2</vcpu> > <features> > <acpi/> > <apic/> > </features> > <os> > <type arch='x86_64' machine='q35'>hvm</type> > </os> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>restart</on_crash> > <devices> > <disk type='volume' device='disk'> > <driver name='qemu' type='raw'/> > <source pool='default' volume='converted-sda'/> > <target dev='vda' bus='virtio'/> > </disk> > <interface type='bridge'> > <source bridge='virbr0'/> > <model type='virtio'/> > <mac address='52:54:00:5c:cc:f8'/> > </interface> > <video> > <model type='vga' vram='16384' heads='1'/> > </video> > <graphics type='spice' autoport='yes'/> > <sound model='ich6'/> > <rng model='virtio'> > <backend model='random'>/dev/urandom</backend> > </rng> > <memballoon model='virtio'/> > <panic model='isa'> > <address type='isa' iobase='0x505'/> > </panic> > + <vsock model='virtio'/> > <input type='tablet' bus='usb'/> > <input type='mouse' bus='ps2'/> > <console type='pty'/> > </devices> > </domain> I also checked the device with "lspci", and the module with "lsmod", in the converted domain. Fixes: 05f780c16f0135c657615520c2245b42de1efc3e Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2070186 Signed-off-by: Laszlo Ersek <lersek(a)redhat.com> --- convert/linux_kernels.ml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/convert/linux_kernels.ml b/convert/linux_kernels.ml index 6dead217ef71..307de5729e3e 100644 --- a/convert/linux_kernels.ml +++ b/convert/linux_kernels.ml @@ -61,8 +61,9 @@ let print_kernel_info chan prefix ki = fpf "virtio: blk=%b net=%b rng=%b balloon=%b\n" ki.ki_supports_virtio_blk ki.ki_supports_virtio_net ki.ki_supports_virtio_rng ki.ki_supports_virtio_balloon; - fpf "pvpanic=%b xen=%b debug=%b\n" - ki.ki_supports_isa_pvpanic ki.ki_is_xen_pv_only_kernel ki.ki_is_debug + fpf "pvpanic=%b vsock=%b xen=%b debug=%b\n" + ki.ki_supports_isa_pvpanic ki.ki_supports_virtio_socket + ki.ki_is_xen_pv_only_kernel ki.ki_is_debug let rex_ko = PCRE.compile "\\.k?o(?:\\.xz)?$" let rex_ko_extract = PCRE.compile "/([^/]+)\\.k?o(?:\\.xz)?$" @@ -248,7 +249,7 @@ let detect_kernels (g : G.guestfs) inspect family bootloader = let supports_isa_pvpanic = kernel_supports "pvpanic" "PVPANIC" in let supports_virtio_socket = - kernel_supports "virtio_socket" "VIRTIO_SOCKET" in + kernel_supports "vmw_vsock_virtio_transport" "VIRTIO_VSOCKETS" in let is_xen_pv_only_kernel = check_config "X86_XEN" config_file || check_config "X86_64_XEN" config_file in -- 2.19.1.3.g30247aa5d201

2 3

Exporting ttyACM0 via nbd
by Marco Marinello 31 Mar '22

31 Mar '22

Hi all, I'm trying to export the serial and i2c interfaces of a RaspberryPi to another Linux host. The server OS is Raspbian stretch with nbdkit 1.1.12. If required, I can compile try to compile / backport a newer version on my own. I tried with /usr/bin/nbdkit --newstyle -g dialout -v file file=/dev/ttyACM0 but it says nbdkit: error: file is not regular or block device: /dev/ttyACM0 Also streaming does not seem to fit to my use case since I will need bi-directional communication. Is what I'm trying to do possible? Can you give me some hint / reference to look into? Thanks in advance, All the best Marco

2 1

inspect-os cannot detect Windows hostname
by Nikolay Ivanets 30 Mar '22

30 Mar '22

Hello I've faced several different cases where inspect-os cannot detect Windows hostname. I see the following message during inspect-os call (in debug mode): check_windows_system_registry: cannot locate HKLM\SYSTEM\ControlSet001\Services\Tcpip\Parameters and/or Hostname key inspect-get-hostname returns 'unknown' in such case: ><fs> inspect-get-hostname /dev/sda1 unknown However, hostname can be read using: ><fs> mount /dev/sda1 / ><fs> debug sh 'hivexget /sysroot/Windows/System32/config/SYSTEM ControlSet001\\Services\\Tcpip\\Parameters hostname' MidServerNew A sample registry hive is attached. -- Mykola Ivanets

1 0

Regression with LDM devices
by Nikolay Ivanets 30 Mar '22

30 Mar '22

Hello This commit caused a regression with LDM devices: https://github.com/libguestfs/libguestfs/commit/86577ee3883836c1c4fff258c05… 1. list-filesystems double all LDM volumes: ><fs> list-filesystems /dev/mapper/ldm_vol_AQ-W10-EFI-VM-Dg0_Volume5: ntfs /dev/sda1: ntfs /dev/sda2: vfat /dev/mapper/ldm_vol_AQ-W10-EFI-VM-Dg0_Volume5: ntfs 2. inspect-os in this case detects 2 OSes: ><fs> inspect-os /dev/mapper/ldm_vol_AQ-W10-EFI-VM-Dg0_Volume5 /dev/mapper/ldm_vol_AQ-W10-EFI-VM-Dg0_Volume5 The issue is that now we explicitly adds DM devices to a list of file systems, but latter we also selectively adds LDM volumes. Which also DM devices: ><fs> list-ldm-volumes /dev/mapper/ldm_vol_AQ-W10-EFI-VM-Dg0_Volume5 ><fs> list-dm-devices /dev/mapper/ldm_vol_AQ-W10-EFI-VM-Dg0_Volume5 It would be easy to fix leaving only unique file systems eliminating duplicates, however it would not help if LDM contains volumes of type other than simple. Such a volumes consist of LDM partitions which are also DM devices and thus the mentioned patch will include them into a list if file systems but LDM partitions cannot contains file system. Here you can see an example of both cases where list-filesystems returns doubled LDM volumes as well as LDM partitions. Later cannot contains file system: ><fs> list-filesystems /dev/mapper/ldm_part_WIN-RSVUTBNGR5D-Dg0_Disk1-01: unknown /dev/mapper/ldm_part_WIN-RSVUTBNGR5D-Dg0_Disk1-02: ntfs /dev/mapper/ldm_part_WIN-RSVUTBNGR5D-Dg0_Disk2-01: unknown /dev/mapper/ldm_part_WIN-RSVUTBNGR5D-Dg0_Disk2-02: ntfs /dev/mapper/ldm_part_WIN-RSVUTBNGR5D-Dg0_Disk3-01: unknown /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume1: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume10: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume11: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume2: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume3: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume4: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume5: vfat /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume6: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume7: vfat /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume8: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume9: unknown /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume1: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume10: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume11: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume2: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume3: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume4: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume5: vfat /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume6: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume7: vfat /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume8: ntfs /dev/mapper/ldm_vol_WIN-RSVUTBNGR5D-Dg0_Volume9: unknown -- Mykola Ivanets

3 2

[PATCH p2v] ssh: Allow interop with virt-v2v 2.0
by Richard W.M. Jones 29 Mar '22

29 Mar '22

The code previously checked that the remote version of virt-v2v was 1.x and >= 1.28. Since virt-v2v 2.0 has been released and is broadly compatible, allow 2.0 as well. --- ssh.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/ssh.c b/ssh.c index ce0caaa..7ac43eb 100644 --- a/ssh.c +++ b/ssh.c @@ -1030,11 +1030,11 @@ add_output_driver (const char *name, size_t len) static int compatible_version (const char *v2v_version) { - unsigned v2v_minor; + unsigned v2v_major, v2v_minor; - /* The major version must always be 1. */ - if (!STRPREFIX (v2v_version, "1.")) { - set_ssh_error ("virt-v2v major version is not 1 (\"%s\"), " + /* The major version must always be 1 or 2. */ + if (! (STRPREFIX (v2v_version, "1.") || STRPREFIX (v2v_version, "2."))) { + set_ssh_error ("virt-v2v major version is not 1 or 2 (\"%s\"), " "this version of virt-p2v is not compatible.", v2v_version); return 0; @@ -1045,13 +1045,13 @@ compatible_version (const char *v2v_version) * that we published during development, nor (b) using old virt-v2v. * We should remain compatible with any virt-v2v after 1.28. */ - if (sscanf (v2v_version, "1.%u", &v2v_minor) != 1) { + if (sscanf (v2v_version, "%u.%u", &v2v_major, &v2v_minor) != 2) { set_ssh_internal_error ("cannot parse virt-v2v version string (\"%s\")", v2v_version); return 0; } - if (v2v_minor < 28) { + if (v2v_major == 1 && v2v_minor < 28) { set_ssh_error ("virt-v2v version is < 1.28 (\"%s\"), " "you must upgrade to virt-v2v >= 1.28 on " "the conversion server.", v2v_version); -- 2.35.1

2 4

[p2v PATCH 0/2] recognize NVMe devices
by Laszlo Ersek 28 Mar '22

28 Mar '22

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1945835 The first (trivial) patch modifies the code, the second (cumbersome) patch adds a new makefile target to exercise the code. I separated the patches like this so that the second one can be dropped if necessary. Thanks, Laszlo Laszlo Ersek (2): recognize NVMe devices Makefile.am: add "run-virt-p2v-in-an-nvme-vm" Makefile.am | 32 ++++++++++++++++++++ main.c | 1 + docs/p2v-hacking.pod | 11 ++++--- 3 files changed, 40 insertions(+), 4 deletions(-) -- 2.19.1.3.g30247aa5d201

2 4

[PATCH v2 0/3] Spec cleanups before 64-bit extensions
by Eric Blake 25 Mar '22

25 Mar '22

Widening the cc list this time. Patch 1 and 2 are unchanged, but I spent more time rewording the changes in patch 3. Eric Blake (3): docs: Clarify NBD_REPLY_TYPE_ERROR lengths docs: Clarify structured reads vs. block size constraints spec: Clarify BLOCK_STATUS reply details doc/proto.md | 57 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 22 deletions(-) -- 2.35.1

2 7

Virt-v2v binary size
by Richard W.M. Jones 24 Mar '22

24 Mar '22

... is about 5MB stripped, which is not terrible, but also not great. I wrote a trivial Perl script to try to find out what's using the space and the report is attached. Also: $ size -d v2v/virt-v2v text data bss dec hex filename 3580677 1654384 36944 5272005 5071c5 v2v/virt-v2v Main take aways are: Frametables take around 21% of the on-disk space and around 31% of the text, which is a bit surprising. These are added by the OCaml compiler in order to track which stack slots are GC roots. I think the OCaml compiler could do a lot better here and I might raise an upstream bug if I can come up with a small demonstrator. However simply reducing the amount of code linked in helps. Libguestfs bindings are the next big item. Virt-v2v links dynamically to libguestfs.so.0, but the bindings code that translates from OCaml to C and back is linked statically, and it's huge. It's particularly big because we link in every binding (libguestfs has something like 650 entry points), but we only use a handful from virt-v2v. I think we might be able to do something about this. Some OCaml standard libraries are inexplicably big (Scanf? List?!) Most of the rest is actual code in virt-v2v or common/ subdirectory code. Code in particular modules is quite compact eg. Convert_linux which does the bulk of conversion of Linux guests is 24K. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

2 2