[PATCH] daemon: inspect_fs_windows: Handle parted errors
by Sam Eiderman
By creating an empty disk and using it as the first disk of the vm (i.e.
/dev/sda, /dev/sdb{1,2} contains the windows fses) we change the
iteration order of the disks.
This causes inspect_os() to fail since Parted returns a Unix_error if
the device does not contain any partitions - fix this by handling this
Unix_error.
Signed-off-by: Sam Eiderman <sameid(a)google.com>
---
daemon/inspect_fs_windows.ml | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/daemon/inspect_fs_windows.ml b/daemon/inspect_fs_windows.ml
index c4a05bc38..bc6b98b60 100644
--- a/daemon/inspect_fs_windows.ml
+++ b/daemon/inspect_fs_windows.ml
@@ -365,8 +365,10 @@ and map_registry_disk_blob_mbr devices blob =
let device =
List.find (
fun dev ->
- Parted.part_get_parttype dev = "msdos" &&
- pread dev 4 0x01b8 = diskid
+ try
+ Parted.part_get_parttype dev = "msdos" &&
+ pread dev 4 0x01b8 = diskid
+ with Unix.Unix_error (Unix.EINVAL, _, _) -> false
) devices in
(* Next 8 bytes are the offset of the partition in bytes(!) given as
@@ -402,7 +404,10 @@ and map_registry_disk_blob_gpt partitions blob =
fun part ->
let partnum = Devsparts.part_to_partnum part in
let device = Devsparts.part_to_dev part in
- let typ = Parted.part_get_parttype device in
+ let typ =
+ try
+ Parted.part_get_parttype device
+ with Unix.Unix_error (Unix.EINVAL, _, _) -> "unknown" in
if typ <> "gpt" then false
else (
let guid = Parted.part_get_gpt_guid device partnum in
--
2.27.0.212.ge8ba1cc988-goog
3 years, 7 months
[PATCH v2v] Add ALT support
by Richard W.M. Jones
Patch supplied by Mikhail Gordeev, posting for review.
I have compile tested it and checked the code and it looks all
fine to me, so ACK from my point of view. I did not actually
run it because I don't have an ALT Linux install, but it
doesn't seem as if it would affect any other distro.
Rich.
3 years, 8 months
[PATCH nbdkit NOT WORKING 0/2] vddk: Relax threading model.
by Richard W.M. Jones
I believe this roughly implements Nir's proposal here:
https://www.redhat.com/archives/libguestfs/2020-August/msg00028.html
Unfortunately it doesn't work for me. It actually slows things down
quite a lot, for reasons I don't understand. Note the adjustment of
the pool-max parameter and how it affects the total time. The results
are quite reproducible.
$ ./nbdkit -r -U - vddk libdir=~/tmp/vddk-7.0.0/vmware-vix-disklib-distrib user=root password=*** server=*** thumbprint=*** vm=moref=3 file='[datastore1] Fedora 28/Fedora 28.vmdk' pool-max=1 --run 'time qemu-img convert -p -W -m 16 $nbd /var/tmp/out'
(100.00/100%)
real 1m8.031s
user 0m0.112s
sys 0m1.560s
$ ./nbdkit -r -U - vddk libdir=~/tmp/vddk-7.0.0/vmware-vix-disklib-distrib user=root password=*** server=*** thumbprint=*** vm=moref=3 file='[datastore1] Fedora 28/Fedora 28.vmdk' pool-max=8 --run 'time qemu-img convert -p -W -m 16 $nbd /var/tmp/out'
(100.00/100%)
real 1m27.790s
user 0m0.099s
sys 0m1.625s
I'm still investigating this. Could be a simple coding error somewhere.
Rich.
3 years, 8 months
Weird results from g.sh()
by Sam Eiderman
Hi,
I'm using libguestfs 1.42,
When I run the following python3 commands on a rhel7.8:
print(g.ls('/sys'))
print(g.sh('ls /sys'))
I get:
[]
block
bus
class
dev
devices
firmware
fs
hypervisor
kernel
module
power
It seems that g.ls('/sys') is chrooted correctly.
But g.sh('ls /sys') isn't.
I came across this behavior when I used g.command(['grub2-mkconfig', '-o',
'....']) on rhel7.8 and it didn't produce the correct linuxefi/initrdefi
entries in grub, since in rhel7.8 the uefi detection algorithm is checking
in bash '[ -d /sys/firmware/efi ]'.
https://src.fedoraproject.org/rpms/grub2/blob/71e0fb4ea646071ec86e4a65bb2...
I thought I could simply g.mkdir_p('/sys/firmware/efi') - but that created
a fake dir in my chrooted environment, where grub2-mkconfig somehow was
inspecting the real /sys as demonstrated in the above 'ls' example.
Any ideas on why sh jailbreaks the chroot? is /sys set through some
environment variable?
Thanks!
3 years, 8 months
[nbdkit PATCH 0/2] ext2 export list tweaks
by Eric Blake
Applies on top of my pending series for the exportname filter,
addressing one of the todo's in that cover letter.
Eric Blake (2):
filters: Add .export_description wrappers
ext2: Supply .list_exports and .default_export
filters/ext2/nbdkit-ext2-filter.pod | 3 +-
tests/Makefile.am | 16 +++-
filters/ext2/ext2.c | 125 +++++++++++++++++++---------
filters/gzip/gzip.c | 40 ++++++---
filters/partition/partition.c | 58 +++++++++----
filters/tar/tar.c | 48 +++++++----
filters/xz/xz.c | 43 +++++++---
tests/test-ext2-exportname.sh | 73 ++++++++++++++++
8 files changed, 305 insertions(+), 101 deletions(-)
create mode 100755 tests/test-ext2-exportname.sh
--
2.28.0
3 years, 8 months
[nbdkit PATCH v2 0/8] exportname filter
by Eric Blake
This is a revision of my .default_export work, plus new work on
.export_descriptions and a new exportname filter. I think it is
now ready to check in.
Things I'd still like in 1.22:
- the file plugin should implement .list_exports (patch already posted,
but it needs rebasing on this series)
- the ext2 filter should override .list_exports when in exportname mode
- the nbd plugin should be able to use libnbd 1.4 features for .list_exports
- more language bindings for new functions (perl, python, ocaml, ...)
nice to have, but can be post-release:
- the file plugin in dir= mode should have default=none|largest (default
behavior none says "" is an error because it is not a file, setting it
to largest picks the largest file in the directory for "")
- the tar filter should consider an exportname mode
- figure out how to let filters open the plugin without a current
connection into the filter
Eric Blake (8):
api: Add .default_export
api: Add nbdkit_add_default_export
server: Respond to NBD_INFO_NAME request
api: Add nbdkit_str[n]dup_intern helper
sh, eval: Implement .default_export
api: Alter .list_exports
api: Add .export_description
exportname: New filter
docs/nbdkit-filter.pod | 56 ++-
docs/nbdkit-plugin.pod | 123 ++++++-
docs/nbdkit-protocol.pod | 14 +
.../exportname/nbdkit-exportname-filter.pod | 154 ++++++++
filters/ext2/nbdkit-ext2-filter.pod | 5 +
filters/log/nbdkit-log-filter.pod | 2 +-
plugins/eval/nbdkit-eval-plugin.pod | 6 +
plugins/file/nbdkit-file-plugin.pod | 9 +-
plugins/sh/nbdkit-sh-plugin.pod | 39 +-
include/nbdkit-common.h | 6 +
include/nbdkit-filter.h | 13 +-
include/nbdkit-plugin.h | 4 +-
configure.ac | 2 +
filters/exportname/Makefile.am | 67 ++++
tests/Makefile.am | 8 +-
server/internal.h | 22 +-
server/backend.c | 93 +++--
server/connections.c | 2 +-
server/exports.c | 32 +-
server/filters.c | 31 +-
server/main.c | 26 +-
server/nbdkit.syms | 3 +
server/plugins.c | 44 ++-
server/protocol-handshake-newstyle.c | 87 ++++-
server/public.c | 48 +++
server/test-public.c | 9 +-
plugins/sh/methods.h | 2 +
plugins/cc/cc.c | 73 ++--
plugins/eval/eval.c | 70 ++--
plugins/ondemand/ondemand.c | 11 +-
plugins/sh/methods.c | 84 ++++-
plugins/sh/sh.c | 70 ++--
plugins/sh/example.sh | 2 +-
filters/exportname/exportname.c | 342 ++++++++++++++++++
filters/log/log.c | 18 +-
filters/tls-fallback/tls-fallback.c | 60 ++-
tests/test-eval-exports.sh | 34 +-
tests/test-export-info.sh | 111 ++++++
tests/test-exportname.sh | 184 ++++++++++
tests/test-tls-fallback.sh | 53 ++-
tests/test-layers-filter.c | 13 +-
tests/test-layers-plugin.c | 8 +
tests/test-layers.c | 24 +-
TODO | 20 +-
44 files changed, 1790 insertions(+), 294 deletions(-)
create mode 100644 filters/exportname/nbdkit-exportname-filter.pod
create mode 100644 filters/exportname/Makefile.am
create mode 100644 filters/exportname/exportname.c
create mode 100755 tests/test-export-info.sh
create mode 100755 tests/test-exportname.sh
--
2.28.0
3 years, 8 months
[nbdkit PATCH 0/3] .list_exports in nbd plugin
by Eric Blake
Another series on top of my exportname filter, marking off another
todo bullet point. With this, you can now use the NBD plugin as a
transparent passthrough of all export names served by the remote
server in both directions (list advertisement server to client, and
export name from client to server).
Eric Blake (3):
nbd: Implement .default_export, .export_description
nbd: Add dynamic-export=true option
nbd: Implement .list_exports
plugins/nbd/nbdkit-nbd-plugin.pod | 30 +++-
tests/Makefile.am | 4 +
plugins/nbd/nbd.c | 250 +++++++++++++++++++++++++-----
tests/test-nbd-dynamic-content.sh | 75 +++++++++
tests/test-nbd-dynamic-list.sh | 162 +++++++++++++++++++
5 files changed, 482 insertions(+), 39 deletions(-)
create mode 100755 tests/test-nbd-dynamic-content.sh
create mode 100755 tests/test-nbd-dynamic-list.sh
--
2.28.0
3 years, 8 months
nbdkit, Windows, Wine, scripts and tests
by Richard W.M. Jones
I spent a bit of my holiday trying to get some more tests working on
the Windows port. Unfortunately I hit some unexpected and deep snags.
I'm writing down these thoughts before I forget them.
nbdkit runs shell scripts from a lot of places. The places which are
important for tests are:
(a) nbdkit --run 'script'.
(b) linuxdisk plugin (used to create tests/disk, so important for
tests) runs "du" and "mke2fs -d" commands.
On Unix-like platforms these use system(3), and they also do in the
Windows port since Win32 also has system(). When running on actual
Windows the commands are run through CMD.EXE as you'd expect. Of
course Windows doesn't have the rich and powerful shell environment
that Unix has so there's no way for nbdkit --run to work exactly the
same on Windows as it does on Unix, but that's all to be expected.
More interesting for running the tests is what happens when running
under Wine on Linux.
I imagined that Linux -> Wine nbdkit.exe -> system ("qemu-img ...")
would work by the Windows process running the Unix command directly.
Unfortunately that's not how it works at all. In fact Wine emulates
system() using CMD.EXE, and so none of the tests as written will work.
It's possible to get system() under Wine to run Unix commands, but
complicated to say the least: https://stackoverflow.com/a/45545068
However even if we get that lot to work, there's still a big catch.
Paths are generally rewritten using nbdkit_realpath which on Windows
returns paths like "C:\Windows\Temp\socket". Wheneven these appear in
variables (eg. $unixsocket, $uri, and variables passed to the
linuxdisk plugin subcommands) these will make no sense to the Unix
commands we want to run.
In summary, a test which does:
nbdkit .. --run 'qemu-img info "$uri"'
will currently try to run:
CMD.EXE /C qemu-img info 'nbd+unix:///?socket=C:\Windows\Temp\sock'
[Note that on actual Windows, assuming qemu-img.exe is available, I
would expect this to work, although we still need to adjust how
variables are set in the script fragment we pass to system(). The
problem is not how to get this to work on Windows, but how to get it
to work under Wine on Linux so we can run more tests.]
Even if we resolve the problem with running shell commands, it will
still not work because it will run:
qemu-img info 'nbd+unix:///?socket=C:\Windows\Temp\sock'
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3 years, 8 months
Unknown libguestfs failure / race
by Sam Eiderman
Hi,
I am getting a weird libguestfs failure from time to time:
2020-08-27 08:48:40 T libguestfs - 0 - appliance - SeaBIOS (version 1.12.0-1)
2020-08-27 08:48:40 T libguestfs - 0 - appliance - Booting from ROM...
2020-08-27 08:48:40 T libguestfs - 0 - library - child_cleanup:
0xd5a840: child process died
Let me elaborate more about my environment
I am running libguestfs 1.42.0 on GCP - that's nested virtualization.
Using python bindings.
Inside a docker.
I wrote an output hook in python to emit logs more nicely.
The supermin image is debian 10.
The failure seems to happen from time to time just as I launch 16
"libguestfs"s on my 16 core machine.
(This is a testing framework, so we are using libguestfs here to
inspect and change different workloads, very similar to virt-v2v)
Here is the full log of the sporadic failure:
2020-08-27 08:48:38 T libguestfs - 0 - enter - set_memsize
2020-08-27 08:48:38 T libguestfs - 0 - enter - add_drive
2020-08-27 08:48:38 T libguestfs - 0 - enter - launch
2020-08-27 08:48:38 T libguestfs - 0 - enter - max_disks
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_tmpdir
2020-08-27 08:48:38 T libguestfs - 0 - enter - version
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_backend
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: program=my-v2v
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: version=1.42.0
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: backend
registered: unix
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: backend registered: uml
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: backend
registered: libvirt
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: backend
registered: direct
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: backend=direct
2020-08-27 08:48:38 T libguestfs - 0 - library - launch:
tmpdir=/tmp/libguestfszNoLuV
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: umask=0022
2020-08-27 08:48:38 T libguestfs - 0 - library - launch: euid=0
2020-08-27 08:48:38 T libguestfs - 0 - library - begin testing qemu features
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_cachedir
2020-08-27 08:48:38 T libguestfs - 0 - library - checking for
previously cached test results of /usr/bin/kvm, in /var/tmp/.guestfs-0
2020-08-27 08:48:38 T libguestfs - 0 - library - loading previously
cached test results
2020-08-27 08:48:38 T libguestfs - 0 - library - qemu version: 3.1
2020-08-27 08:48:38 T libguestfs - 0 - library - qemu mandatory locking: yes
2020-08-27 08:48:38 T libguestfs - 0 - library - qemu KVM: disabled
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_backend_setting
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_sockdir
2020-08-27 08:48:38 T libguestfs - 0 - library - finished testing qemu features
2020-08-27 08:48:38 T libguestfs - 0 - enter - get_backend_setting
2020-08-27 08:48:38 T libguestfs - 0 - library - command: run: dmesg |
grep -Eoh 'lpj=[[:digit:]]+'
2020-08-27 08:48:38 T libguestfs - 0 - library - read_lpj_from_dmesg:
calculated lpj=8800000
2020-08-27 08:48:38 T libguestfs - 0 - library - command: run: file
2020-08-27 08:48:38 T libguestfs - 0 - library - command: run: \ --
/appliance/root
2020-08-27 08:48:38 T libguestfs - 0 - appliance - /usr/bin/kvm \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -global
virtio-blk-pci.scsi=off \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -no-user-config \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -enable-fips \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -nodefaults \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -display none \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -machine accel=kvm:tcg \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -m 2048 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -no-reboot \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -rtc driftfix=slew \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -no-hpet \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -global
kvm-pit.lost_tick_policy=discard \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -kernel
/appliance/kernel \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -initrd
/appliance/initrd \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -object
rng-random,filename=/dev/urandom,id=rng0 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
virtio-rng-pci,rng=rng0 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
virtio-scsi-pci,id=scsi \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -drive
file=/tmp/xww-a1dtzg8i.qcow2,cache=writeback,format=qcow2,id=hd0,if=none
\
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
scsi-hd,drive=hd0 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -drive
file=/appliance/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw
\
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
scsi-hd,drive=appliance \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
virtio-serial-pci \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -serial stdio \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device sga \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -chardev
socket,path=/tmp/libguestfsISN6Xx/guestfsd.sock,id=channel0 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -device
virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
2020-08-27 08:48:38 T libguestfs - 0 - appliance - -append
"panic=1 console=ttyS0 edd=off udevtimeout=6000
udev.event-timeout=6000 no_timer_check lpj=8800000 printk.time=1
cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable
8250.nr_uarts=1 root=UUID=dd6872eb-d621-499d-be42-64d78754c255
selinux=0 guestfs_verbose=1 TERM=linux vsyscall=emulate"
2020-08-27 08:48:40 T libguestfs - 0 - library - responding to serial
console Device Status Report
2020-08-27 08:48:40 T libguestfs - 0 - appliance - ^[[1;256r^[[256;256H^[[6n
2020-08-27 08:48:40 T libguestfs - 0 - appliance - Google, Inc.
2020-08-27 08:48:40 T libguestfs - 0 - appliance - Serial Graphics
Adapter 03/05/17
2020-08-27 08:48:40 T libguestfs - 0 - appliance - SGABIOS $Id:
sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (generic@generic) Sun, 05
Mar 2017 16:09:17 +0100
2020-08-27 08:48:40 T libguestfs - 0 - appliance - Term: 80x24
2020-08-27 08:48:40 T libguestfs - 0 - appliance - 4 0
2020-08-27 08:48:40 T libguestfs - 0 - appliance - ^[[2J
2020-08-27 08:48:40 T libguestfs - 0 - appliance - SeaBIOS (version 1.12.0-1)
2020-08-27 08:48:40 T libguestfs - 0 - appliance - Booting from ROM...
2020-08-27 08:48:40 T libguestfs - 0 - library - child_cleanup:
0xd5a840: child process died
2020-08-27 08:48:40 T libguestfs - 0 - library - sending SIGTERM to process 459
2020-08-27 08:48:40 T libguestfs - 0 - library - qemu maxrss 60136K
2020-08-27 08:48:40 T libguestfs - 0 - subprocess_quit
2020-08-27 08:48:40 T libguestfs - 0 - library - closing guestfs
handle 0xd5a840 (state 0)
2020-08-27 08:48:40 T libguestfs - 0 - close
2020-08-27 08:48:40 T libguestfs - 0 - library - command: run: rm
2020-08-27 08:48:40 T libguestfs - 0 - library - command: run: \ -rf
/tmp/libguestfszNoLuV
2020-08-27 08:48:40 T libguestfs - 0 - library - command: run: rm
2020-08-27 08:48:40 T libguestfs - 0 - library - command: run: \ -rf
/tmp/libguestfsISN6Xx
Usually everything is working and the output is:
2020-08-27 07:41:10 T libguestfs - 0 - appliance - SeaBIOS (version 1.12.0-1)
2020-08-27 07:41:11 T libguestfs - 0 - appliance - Booting from ROM...
2020-08-27 07:41:12 T libguestfs - 0 - appliance - ^[[2J[ 0.000000]
Linux version 4.19.0-10-amd64 (debian-kernel(a)lists.debian.org) (gcc
version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.132-1 (2020-07-24)
2020-08-27 07:41:12 T libguestfs - 0 - appliance - [ 0.000000]
Command line: panic=1 console=ttyS0 edd=off udevtimeout=6000
udev.event-timeout=6000 no_timer_check lpj=8800000 printk.time=1
cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable
8250.nr_uarts=1 root=UUID=dd6872eb-d621-499d-be42-64d78754c255
selinux=0 guestfs_verbose=1 TERM=linux vsyscall=emulate
It seems that the qemu process has died somewhere between the ROM boot
and the kernel loading.
Any ideas how to debug this?
Thanks!
Sam
3 years, 8 months
ANNOUNCE: nbdkit 1.22 - high performance NBD server
by Richard W.M. Jones
I'm pleased to announce the release of nbdkit 1.22, a high performance
plugin-based Network Block Device (NBD) server.
https://en.wikipedia.org/wiki/Network_block_device
Key features of nbdkit:
* Multithreaded NBD server written in C with good performance.
* Minimal dependencies for the basic server.
* Liberal license (BSD) allows nbdkit to be linked to proprietary
libraries or included in proprietary code.
* Well-documented, simple plugin API with a stable ABI guarantee.
Lets you export “unconventional” block devices easily.
* You can write plugins in C, Go, Lua, Perl, Python, OCaml, Ruby,
Rust, shell script or Tcl.
* Filters can be stacked in front of plugins to transform the output.
Git: https://github.com/libguestfs/nbdkit
Download: https://download.libguestfs.org/nbdkit/1.22-stable/
Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=16469
Debian: https://packages.debian.org/nbdkit
Release notes (http://libguestfs.org/nbdkit-release-notes-1.22.1.html)
These are the release notes for nbdkit stable release 1.22. This
describes the major changes since 1.20.
nbdkit 1.22.0 was released on 27 August 2020.
Security
There were no security issues found. All past security issues and
information about how to report new ones can be found in
nbdkit-security(1).
Plugins
New nbdkit-cdi-plugin(1) (Containerized Data Importer) lets you export
a single layer from a container image, allowing you to access disk
images distributed using container registry infrastructure.
New nbdkit-ondemand-plugin(1) creates filesystems for clients on
demand.
New nbdkit-torrent-plugin(1) serves a BitTorrent file or magnet link
over NBD.
nbdkit-data-plugin(1) and nbdkit-memory-plugin(1) have a new
"allocator" parameter giving more fine-grained control over the
allocation strategy. Currently implemented strategies are: sparse
array, sparse array with zstd compression, malloc, and malloc with
mlock.
nbdkit-data-plugin(1) data format now supports strings and alignment.
The "disk2data.pl" script has been enhanced to detect repeated
sequences and produce a more compact output.
nbdkit-curl-plugin(1) adds support for arbitrary custom request
headers. Also it now supports scripts for generating cookies and
custom request headers. This is especially useful when accessing web
servers that require authentication which has to be renewed
periodically.
nbdkit-curl-plugin(1) now supports HTTP/2 servers (David Edmondson,
Pino Toscano, Zi Liu).
nbdkit-curl-plugin(1) adds support for the "User-Agent" header.
nbdkit-file-plugin(1) can export multiple files in a directory as
separate exports (Eric Blake).
nbdkit-file-plugin(1) supports new "cache=none" and
"fadvise=normal|random|sequential" parameters which can be used to
optimize access patterns and avoid filling the page cache.
nbdkit-nbd-plugin(1) supports proxying to another NBD server over
"AF_VSOCK" (Eric Blake).
nbdkit-nbd-plugin(1) supports new "command=" and "socket-fd="
parameters which allow you to run a subordinate NBD server as a command
directly from the plugin, or pass in an already connected NBD endpoint.
nbdkit-streaming-plugin(1) supports streaming to and from local pipes.
nbdkit-vddk-plugin(1) now has support for VDDK 7.0. A new and useless
"error" message printed by VDDK 7 has been suppressed (thanks Ming Xie,
Pino Toscano).
nbdkit-ext2-plugin(1) has been removed. It was deprecated in
nbdkit 1.17.8. All existing uses can be replaced with
nbdkit-ext2-filter(1).
The "-" character is now permitted within plugin and filter names (Eric
Blake).
Filters
New nbdkit-ddrescue-filter(1) lets you test disk images recovered by
ddrescue(1) (François Revol).
New nbdkit-gzip-filter(1) can decompress a plugin that contains gzip-
compressed data. It replaces nbdkit-gzip-plugin(1) which is deprecated
and will be removed in nbdkit 1.26.
New nbdkit-pause-filter(1) can temporarily stop and later resume NBD
requests. This can be used when taking snapshots.
New nbdkit-swab-filter(1) swaps bytes in the underlying plugin, which
can be used to adjust images that have an incorrect byte order
(François Revol).
New nbdkit-tar-filter(1) reads and writes files inside a tar file in
the underlying plugin. It replaces nbdkit-tar-plugin(1) which is
deprecated and will be removed in nbdkit 1.26.
New nbdkit-tls-fallback-filter(1) provides alternate content when a
client connects without using TLS to a server that permits but does not
require TLS (Eric Blake).
nbdkit-fua-filter(1) has new option "fuamode=pass" to pass through FUA
and flush requests unchanged (disabling the filter). A new, unsafe
"fuamode=discard" mode was added which discards FUA and flush requests.
Language bindings
New nbdkit-cc-plugin(3) lets you write small nbdkit plugins as inline
C, C++ or OCaml "scripts" (and potentially other ahead-of-time compiled
programming languages).
nbdkit-rust-plugin(3) bindings have been replaced by higher level and
more idiomatic bindings. These also support a greater range of server
features (Alan Somers).
nbdkit-python-plugin(3) now supports parallel plugins. These cannot
run Python code in parallel (because of the Python GIL), but for Python
code that blocks, eg. calling out to external services, requests can
now be handled in parallel (thanks Nir Soffer).
nbdkit-python-plugin(3) has a greater range of examples (Nir Soffer).
nbdkit-python-plugin(3) adds support for getting the client export
name, calling "nbdkit_debug", and also for extents.
Server
"nbdkit --dump-config" has new fields: "version_major" and
"version_minor" to allow easier version parsing from scripts.
"host_cpu" and "host_os" so you can find out if nbdkit was cross-
compiled for another platform. "zstd=(yes|no)" so you can find out if
nbdkit was compiled with zstd compression support.
The -e/--exportname option (which never did anything useful previously)
is now used to pass the preferred exportname to the --run option. In
general you should never use this option except in very limited and
exceptional circumstances (Eric Blake).
API
New ".after_fork" callback which can be used for plugins which need to
create background threads, since those threads would otherwise be
stranded if the server forks into the background. The VDDK plugin now
defers VDDK library initialization til ".after_fork" because it is
thought that the library creates background threads especially when
accessing remote servers.
New API "nbdkit_extents_aligned" is a helper function for filters that
retrieves extents as needed until at least a certain range is covered
(Eric Blake).
New "nbdkit_is_tls" can be called to determine if TLS was negotiated on
the connection. Plugins could use this to hide certain exports from
non-authenticated/non-encrypted clients (Eric Blake).
"nbdkit_read_password" with "password=-" will now return an error if
the input is not a tty. Also "password=-FD" is disallowed if the file
descriptor is 0, 1 or 2.
Bug fixes
The VDDK plugin was extensively refactored to fix many bugs: Reading
passwords from a tty should work again. The plugin should now work
properly when daemonized or using the --run option. The threading
model has been relaxed to "SERIALIZE_REQUESTS" which should give small
performance improvements when using multiple connections.
The "base64" option of nbdkit-data-plugin(1) has been fixed so it sets
the implicit size correctly.
The "<FILE" formatter of nbdkit-data-plugin(1) now works if you use it
more than once.
Documentation
New nbdkit-client(1) manual page documents how to mount filesystems
from NBD exports on client machines, either directly or using systemd
mountpoints.
Examples now use "NBDKIT_API_VERSION 2" so that the examples match the
documentation.
Tests
The tests should now pass on OpenBSD ≥ 6.4.
tests/Makefile.am was refactored: There is now no longer a single
massive "EXTRA_DIST". Creation of test disks was refactored.
Some tests will now use new libnbd 1.4 tools nbdcopy(1) and nbdinfo(1)
if available, and skip the tests otherwise.
New test functions "requires_plugin" and "requires_filter" allows tests
to be skipped if the corresponding plugin/filter was not compiled on a
particular platform. So tests should now not fail when an optional
external library needed by a plugin/filter is not available.
Build
Honggfuzz may now be used for fuzz testing.
If nbdkit-python-plugin(3) is enabled, Python ≥ 3.6 must be used.
Internals
The export name is now passed down through filter ".open" callbacks,
allowing filters to modify the export name passed through to plugins.
The final thread model is passed to filter ".get_ready" callbacks.
SEE ALSO
nbdkit(1).
AUTHORS
Authors of nbdkit 1.22:
Alan Somers
(7 commits)
Eric Blake
(53 commits)
François Revol
(2 commits)
Nir Soffer
(2 commits)
Richard W.M. Jones
(285 commits)
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3 years, 8 months