[Libguestfs] ANNOUNCE: nbdkit 1.36 and libnbd 1.18 released

I'm pleased to announce new stable releases of nbdkit 1.36 and libnbd 1.18. nbdkit is a Network Block Device (NBD) server with a stable plugin ABI and a permissive license. libnbd is an NBD client library in userspace. Among the major new features are: Rust bindings for libnbd, 64 bit extended headers (NBD protocol extension) support in libnbd, a completely rewritten curl plugin for nbdkit, qcow2 support for nbdkit. Complete release notes are attached below. nbdkit 1.36.0 can be downloaded here: https://download.libguestfs.org/nbdkit/1.36-stable/ libnbd 1.18.0 can be downloaded here: https://download.libguestfs.org/libnbd/1.18-stable/ Release notes for nbdkit 1.36 online: https://libguestfs.org/nbdkit-release-notes-1.36.1.html Release notes for libnbd 1.18 online: https://libguestfs.org/libnbd-release-notes-1.18.1.html Rich. ---------------------------------------------------------------------- nbdkit-release-notes-1.36 - release notes for nbdkit 1.36 These are the release notes for nbdkit stable release 1.36. This describes the major changes since 1.34. nbdkit 1.36.0 was released on 27 September 2023. Security No security issues were identified in this release. All past security issues and information about how to report new ones can be found in nbdkit-security(1). Plugins New nbdkit-ones-plugin(1) which creates a fully allocated disk containing all 0xff (all ones), or another byte of your choice. nbdkit-curl-plugin(1) now uses a curl "multi" interface. This enables much better performance, and also allows the curl plugin to handle requests in parallel. The curl plugin now falls back to making a "GET" request to get the size of the remote file for certain servers which do not support "HEAD" requests. This plugin adds new options: "ipresolve" (force IPv4 or IPv6), "resolve" (force a particular IP address), -D curl.times=1 (print detailed timing stats), and -D curl.verbose.ids=1 (display connection and transfer IDs). nbdkit-memory-plugin(1) now uses a read-write lock to protect internal structures, resulting in improved performance for mostly read workloads. nbdkit-data-plugin(1) now has more optimizations. nbdkit-file-plugin(1) now supports 4k sector sizes on Windows (Brian Carnes). Filters New nbdkit-evil-filter(1) adds random but consistent data corruption to the underlying plugin. New nbdkit-qcow2dec-filter(1) which can decode qcow2 files (but not write to them). nbdkit-ip-filter(1) can now filter by client SELinux label. nbdkit-partition-filter(1) now supports 4k sector sizes (Brian Carnes). nbdkit-retry-request-filter(1) allows the "get_size" operation to be retried. nbdkit-tar-filter(1) adds new "tar-limit" parameter which can be used to ensure the filter does not read indefinite amounts of input when opening the tar file. Filters can now append their own output to nbdkit --dump-plugin output. Language bindings Rust bindings add support for "after_fork", "block_size", "nbdkit_debug", "nbdkit_is_tls", "nbdkit_parse_size", "nbdkit_parse_bool" and "nbdkit_parse_probability". The "open" method can now return an error; note this is not backwards compatible and requires a small source code change to Rust plugins. (Thanks Alan Somers) nbdkit-ocaml-plugin(3) now supports OCaml 5. OCaml bindings add support for "nbdkit_stdio_safe", "nbdkit_is_tls", "nbdkit_peer_name" and "nbdkit_peer_security_context". nbdkit-perl-plugin(3) now supports Perl 5.38. Shell script plugins (nbdkit-sh-plugin(3)) may now ignore stdin in their "pwrite" method, whereas previously it was required to read and discard stdin along error paths (thanks Eric Blake). Server When using the --run option, the default is now to use a private Unix domain socket (as if -U - was specified), whereas in nbdkit ≤ 1.34 the default was to open a TCP port. The new default reflects the most common and safest way to use the --run option. You can find out if nbdkit has the new behaviour by checking "nbdkit --dump-config" and looking for "run_default_socket=Unix" in the output. Debug strings containing control codes and other non-printable characters are now escaped properly. New flag -D nbdkit.environ=1 can be used to dump the server environment in debug output. API New "nbdkit_parse_probability" function which can be used to parse probabilities in various formats, like "10%" or "1:10". New "nbdkit_peer_security_context" function which returns the security context (usually SELinux label) of the client. Bug fixes Fix long-standing double-free in nbdkit-file-plugin(1) when the client closes the connection during negotiation (thanks Eric Blake). nbdkit-file-plugin now has improved error messages when you try to open a file with an empty name (""). Documentation Miscellaneous documentation improvements for Windows (Brian Carnes). Fuzzing instructions updated for Clang 17, AFL++ 4.08c. Tests Ongoing CI maintenance and fixes (Laszlo Ersek, Eric Blake). Many fixes in the test suite for MinGW and Wine. Several internal consistency checks were added to the test suite, to ensure, for example, that we always use "requires_run" when the test uses the nbdkit --run option. Build Optionally "zlib-ng" can be used (instead of "zlib") for faster decompression in various places. No longer link with msvcrt on Windows. "awk" is a build requirement. It was implicitly required before (because it is needed by the "./configure" script generated by autoconf), but now the dependency is explicit (Eric Blake). Internals "podwrapper" has a new --replace parameter which makes it easier to use "AC_SUBST"-substitutions in documentation. SEE ALSO nbdkit(1). AUTHORS Authors of nbdkit 1.36: Alan Somers Brian Carnes Eric Blake Laszlo Ersek Richard W.M. Jones ---------------------------------------------------------------------- libnbd-release-notes-1.18 - release notes for libnbd 1.18 These are the release notes for libnbd stable release 1.18. This describes the major changes since 1.16. libnbd 1.18.0 was released on 27 September 2023. Security Eric Blake found a case where a server could cause libnbd to crash, although not in the normal libnbd configuration. For more information see the announcement here: https://listman.redhat.com/archives/libguestfs/2023-July/032035.html Eric Blake found an issue with nbd_get_size(3) where a server that returns a size > (uint64_t)INT64_MAX and < (uint64_t)-1 would cause nbd_get_size to return a negative number (which is not -1 and so callers may or may not treat it as an error). While no client code in libnbd itself is affected by this, it could affect external clients. libnbd ≥ 1.16.5 now returns an error (-1) and sets nbd_get_errno to "EOVERFLOW" in this case. We have applied for a decision on a CVE. See the announcement here: https://listman.redhat.com/archives/libguestfs/2023-September/032711.html During routine fuzzing we found several security problems which had been introduced during this development cycle and have subsequently been fixed. Stable (even numbered) releases of libnbd should not be vulnerable; do not use the development (odd numbered) releases in production. If you find a security issue, please read SECURITY in the source (online here: https://gitlab.com/nbdkit/libnbd/blob/master/SECURITY). To find out about previous security issues in libnbd, see libnbd-security(3). New APIs nbd_block_status_64(3) nbd_aio_block_status_64(3) Make a 64 bit block status request, see "Protocol" below (Eric Blake). nbd_block_status_filter(3) nbd_aio_block_status_filter(3) Send filtered block status command, see "Protocol" below (Eric Blake). nbd_can_block_status_payload(3) Find out if the server supports filtered block status command (Eric Blake). nbd_set_request_extended_headers(3) nbd_get_request_extended_headers(3) nbd_get_extended_headers_negotiated(3) nbd_opt_extended_headers(3) nbd_aio_opt_extended_headers(3) Set/get whether we request extended headers from the server, and find out if we negotiated extended headers, see "Protocol" below (Eric Blake). Enhancements to existing APIs "qemu:" meta-context constants (eg. "qemu:dirty-bitmap" as "LIBNBD_CONTEXT_QEMU_DIRTY_BITMAP") are now available through the C, Golang, OCaml and Python language bindings (Eric Blake). nbd_shutdown(3) now works correctly when in opt mode (Eric Blake). nbd_set_string(3) adds "LIBNBD_STRICT_AUTO_FLAG" which allows the client to test how servers behave when the payload length flag is adjusted (Eric Blake). Protocol libnbd now supports NBD 64 bit "extended headers" and extent sizes. In practice this allows certain requests such as zeroing very large sections of the disk to be implemented much more efficiently, with servers that support this (Eric Blake). libnbd now supports filtered block status requests (Eric Blake). Tools nbdcopy(1), nbdinfo(1) and nbddump(1) have been expanded to use and report NBD 64 bit / extended header support when the server supports it (Eric Blake). nbdinfo --has can be used as an alias for --can, eg. "nbdinfo --has structured-reply URI" (Eric Blake). nbdinfo makes the export size output optional, for servers which send an oversized one (Eric Blake). nbdcopy now supports "human sizes" for some parameters, eg. "nbdcopy --request-size=1M ...". Language bindings New Rust bindings. There is a basic API for ordinary use, and an asynch API implemented using Tokio. Rust ≥ 1.69 is required. (Tage Johansson, supported by sponsorship from Google Summer of Code 2023, additional review and fixes by Eric Blake). OCaml 5 is now supported. Golang 1.21 is now supported. The minimum version is now 1.17 (Eric Blake). Use "gofmt" to format Golang bindings (Eric Blake). Use "unsafe.Slice" when converting C arrays to Golang slices (Eric Blake). All language bindings support NBD 64 bit / extended headers, and examples of how to use this feature are available (Eric Blake). Tests Fix a couple of race conditions in tests where we did not fully consume stdin in nbdkit-sh-plugin(3) pwrite method (Eric Blake). Fuzzing now disables client-side strictness checks, enabling a wider range of inputs to be fuzzed (Eric Blake). Other improvements and bug fixes Consistently wrap source code at 80 columns (Laszlo Ersek). Debug messages no longer print the very verbose state transitions inside the state machine as these are not usually useful. You can reenable this by defining "-DLIBNBD_STATE_VERBOSE=1" at compile time. Completion ".callback" methods are now always called exactly once, and documentation is clearer on when this happens (Eric Blake). Documentation "podwrapper" has a new --replace parameter which makes it easier to use "AC_SUBST"-substitutions in documentation. Build Automake's "subdir-objects" option is now used (Eric Blake). Multiple, ongoing fixes to the CI tests (Eric Blake). SEE ALSO libnbd(3). AUTHORS Eric Blake Laszlo Ersek Richard W.M. Jones Tage Johansson -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html