[Libguestfs] ANNOUNCE: libnbd 1.8 and nbdkit 1.26 - high performance network block device client and server

I'm pleased to announce the release of nbdkit 1.26, a high performance plugin-based Network Block Device (NBD) server, and libnbd 1.8, a high performance client library. https://en.wikipedia.org/wiki/Network_block_device You can get these from: tarballs: https://download.libguestfs.org/nbdkit/ https://download.libguestfs.org/libnbd/ git: https://gitlab.com/nbdkit Key features of nbdkit: * Multithreaded NBD server written in C with good performance. * Minimal dependencies for the basic server. * Liberal license (BSD) allows nbdkit to be linked to proprietary libraries or included in proprietary code. * Well-documented, simple plugin API with a stable ABI guarantee. Lets you export “unconventional” block devices easily. * You can write plugins in C, Go, Lua, Perl, Python, OCaml, Ruby, Rust, shell script or Tcl. * Filters can be stacked in front of plugins to transform the output. Key features of libnbd: * Synchronous API for ease of use. * Asynchronous API for writing non-blocking, multithreaded clients. You can mix both APIs freely. * High performance. * Minimal dependencies for the basic library. * Well-documented, stable API. * Bindings in several programming languages. * Shell (nbdsh) for command line and scripting. * Copying tool (nbdcopy) for high performance copying and streaming. * Query tool (nbdinfo) to query NBD servers. * FUSE support (nbdfuse) to mount NBD in the local filesystem. Release notes for nbdkit 1.26 nbdkit is now hosted on gitlab: https://listman.redhat.com/archives/libguestfs/2021-February/msg00021.html Security There were no security issues found. All past security issues and information about how to report new ones can be found in nbdkit-security(1). Plugins nbdkit-gzip-plugin and nbdkit-tar-plugin have been removed. They were deprecated in nbdkit 1.22. The functionality has been replaced by nbdkit-gzip-filter(1) and nbdkit-tar-filter(1) respectively. nbdkit-streaming-plugin(1) has been deprecated and will be removed in nbdkit 1.30. You can use nbdcopy(1) instead. nbdkit-curl-plugin(1) adds these features: "followlocation=false" to disable following redirects. "cookiefile" and "cookiejar" to control saving cookies. nbdkit-ssh-plugin(1) now advertises multi-conn if the server is OpenSSH, improving performance (Eric Blake). nbdkit-ondemand-plugin(1) has a new "wait=true" flag allowing you to choose between clients waiting for an export to become free or being immediately rejected. nbdkit-sparse-random-plugin(1) now does not generate fully random data within each block. Using perf we observed that the plugin was spending 34% of the total time during benchmarks just doing this. You can enable the old behaviour using "random-content=true". nbdkit-file-plugin(1) now checks if the file is read-only and exports a read-only NBD connection in this case (Eric Blake). nbdkit-vddk-plugin(1) has been tested with VDDK 7.0.2. Filters There is a new API for use by filters to manage "contexts". Instead of the previous 1-1 relationship between filter connections and plugin connections, filters can now open more than one context into the backend from a single connection, or share contexts between connections (Eric Blake). New nbdkit-multi-conn-filter(1) which allows you to add multi-conn support to plugins which don't support it (although the emulation is expensive), or change whether multi-conn is advertised to clients (Eric Blake). nbdkit-cow-filter(1) now ignores flush/FUA requests (which are not useful for a temporary overlay). The filter now implements extents and efficient trimming. Fine-grained locking is used around shared structures, greatly improving parallel performance. Added support for multi-conn. nbdkit-cache-filter(1) advertises multi-conn when possible (Eric Blake). nbdkit-cache-filter and nbdkit-cow-filter can now handle arbitrary sized disks, instead of rounding the disk size down. nbdkit-blocksize-filter(1) supports parallel requests (Eric Blake). nbdkit-checkwrite-filter(1) supports multi-conn (Eric Blake). nbdkit-ext2-filter(1) no longer supports multi-conn. The filter is single-threaded so it does not matter, but it was not safe. Fix trim, zero and flush support (Eric Blake). nbdkit-readahead-filter(1) fixes multi-conn support (Eric Blake). nbdkit-gzip-filter(1) and nbdkit-xz-filter(1) now both support multi- conn (Eric Blake). Language bindings nbdkit-python-plugin(3) supports the following new features: "after_fork" and "preconnect" methods. "nbdkit.shutdown()" binding. nbdkit-rust-plugin(3) updated for newer Rust compiler versions (Alan Somers). nbdkit-golang-plugin(3) now uses and requires golang modules. Golang ≥ 1.13 must be used. API New ".cleanup" callback for plugins and filters. This is called shortly before ".unload", but unlike ".unload" it is guaranteed that no plugin or filter code has been dlclosed (Eric Blake). New "nbdkit_extents_full" is a helper function for filters which can be used to read all extents from a plugin covering a region. Tests Old plugins that we use for testing were rebuilt so they no longer depend on libselinux (thanks Thomas Weißschuh). nbdkit-ssh-plugin(1) can still be built even if ssh-keygen is unavailable for testing it (thanks Thomas Weißschuh). Improve tests/test-captive.sh to avoid crashes during testing. Test coverage on FreeBSD has been improved, fewer tests should skip unnecessarily. Build nbdkit and plugins can now be built with "-fvisibility=hidden", which allows the compiler to make some optimizations. Add a dependency so nbdkit is relinked if the linker script (nbdkit.syms) changes (Eric Blake). xorriso is now preferred over genisoimage or mkisofs (see https://wiki.debian.org/genisoimage). It is now possible to compile nbdkit with clang with warnings enabled. Several warnings have been suppressed or fixed. "MALLOC_CHECK_" and "MALLOC_PERTURB_" are only enabled when running the tests, not when running nbdkit from the build directory. This makes it easier to do benchmarks. "nbdkit_peer_*" implemented on OpenBSD. Internals When fuzzing nbdkit we now recommend using the "-t 1" flag (which disables threads). This improves the fuzz-stability of nbdkit. AUTHORS Authors of nbdkit 1.26: • Alan Somers • Eric Blake • Michael Ablassmeier • Richard W.M. Jones • Timm Bäder Release notes for libnbd 1.8 libnbd is now hosted on gitlab: https://listman.redhat.com/archives/libguestfs/2021-February/msg00021.html Security If you find a security issue, please read SECURITY in the source (online here: https://gitlab.com/nbdkit/libnbd/blob/master/SECURITY). To find out about previous security issues in libnbd, see libnbd-security(3). There was one security bug found in libnbd during this release cycle. CVE-2021-20286 denial of service when using nbd_set_opt_mode(3) See the full announcement here: https://listman.redhat.com/archives/libguestfs/2021-March/msg00092.html (Found and fixed by Eric Blake). New APIs nbd_get_private_data(3) nbd_set_private_data(3) These calls allow you to store either an unsigned integer or a pointer in the handle for the application to use for its own purposes. nbd_get_uri(3) This call constructs an NBD URI which can be used to connect back to the same server (using nbd_connect_uri(3) or from other tools that support the NBD URI standard). Enhancements to existing APIs Numeric IPv6 URIs (like "nbd://[::1]/") are now supported. libnbd gives a better error message if the server backlog overflows (thanks Xin Long, Lukas Doktor, Eric Blake, Martin Kletzander). New features Continuous integration (CI) tests now run on every commit and merge request to the upstream repository (Martin Kletzander). Tools A great deal of work has been done to improve the performance of nbdcopy(1) (Nir Soffer, Eric Blake). On Linux, nbdcopy now uses readahead when reading from files, and takes steps to preserve the host page cache. nbdcopy new --request-size option to allow controlling the size of requests made to NBD servers. Adjusting this can affect performance (Nir Soffer). nbdcopy now supports "null:" pseudo-target, where it discards the output. This is useful for benchmarking. nbdcopy new -v flag to enable libnbd and other debugging. nbdinfo(1) --list (which lists all exports of a server) no longer exits early if one of the exports is inaccessible or there is a recoverable error (Eric Blake). nbdinfo now prints the URI of each export in the output making it easier to connect back to each export. nbdfuse(1) is now multithreaded and supports issuing parallel NBD commands on a single connection as well as multi-conn, and other enhancements have been made. This resulted in around 50% performance improvement when tested with fio. nbdfuse now supports nbdcopy-style "[ CMD ]", eg: nbdfuse dir/ramdisk [ nbdkit memory 1G ] nbdfuse now supports trimming. Also it supports fast zeroing, but this requires a forthcoming kernel patch before it will work. nbdfuse sets the virtual file permissions appropriately if the NBD export is read-only. nbdfuse new -v flag to enable libnbd and other debugging. Tests Added a test of interoperability with qemu-storage-daemon(1). Note that qemu-storage-daemon was broken in qemu 6.0.0 so this test can fail unless you update to a fixed version (thanks Stefan Hajnoczi, Daniel Berrangé). Added a test of nbdcopy with TLS which was previously untested. Python code style tests are now opt-in using ./configure --enable-python-code-style More tests should now run on FreeBSD and not be skipped. nbdfuse tests now ensure that errors from NBD are transmitted through FUSE back to the local client. "MALLOC_CHECK_" and "MALLOC_PERTURB_" are only enabled when running the tests, not when running libnbd programs from the build directory. This makes it easier to do benchmarks. Other improvements and bug fixes The minimum C compiler is now ISO C99 (previously ANSI C90). FUSE 3 (instead of FUSE 2) is now required to build nbdfuse. The library tries harder not to leak thread-local storage memory on dlclose or exit. (Because of the design of Linux it is not always possible to avoid this, especially in multithreaded programs.) Fix the progress bar in nbdcopy(1). UTF-8 export names are now processed and displayed properly by nbdinfo(1). The --help output of nbdcopy, nbdfuse and nbdinfo now displays a brief summary of all options and has some examples. The generator now only requires ocamlc, not ocamlfind (Abhay Raj Singh). Golang bindings now use and require golang modules. Documentation Document using libnbd with the Meson build system. Added Python examples. Add example of how to integrate libnbd with libev (Nir Soffer). AUTHORS Eric Blake Richard W.M. Jones Martin Kletzander Nir Soffer -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW