[Libguestfs] ANNOUNCE: libnbd 1.6 and nbdkit 1.24 released

I'm pleased to announce the release of nbdkit 1.24, a high performance plugin-based Network Block Device (NBD) server, and libnbd 1.6, a high performance client library. https://en.wikipedia.org/wiki/Network_block_device Key features of nbdkit: * Multithreaded NBD server written in C with good performance. * Minimal dependencies for the basic server. * Liberal license (BSD) allows nbdkit to be linked to proprietary libraries or included in proprietary code. * Well-documented, simple plugin API with a stable ABI guarantee. Lets you export “unconventional” block devices easily. * You can write plugins in C, Go, Lua, Perl, Python, OCaml, Ruby, Rust, shell script or Tcl. * Filters can be stacked in front of plugins to transform the output. Key features of libnbd: * Synchronous API for ease of use. * Asynchronous API for writing non-blocking, multithreaded clients. You can mix both APIs freely. * High performance. * Minimal dependencies for the basic library. * Well-documented, stable API. * Bindings in several programming languages. * Shell (nbdsh) for command line and scripting. * nbdfuse, nbdinfo and nbdcopy tools Release notes for both are attached below. Rich. ------- These are the release notes for nbdkit stable release 1.24. This describes the major changes since 1.22. nbdkit 1.24.0 was released on 7 January 2021. Security There were no security issues found. All past security issues and information about how to report new ones can be found in nbdkit-security(1). Server The server and a subset of the plugins and filters have been ported to Windows using the Win32 API. Only the mingw-w64 toolchain is supported for building nbdkit (not MSVC). You can also cross-compile from Linux and test under Wine. Some features are missing although it is very usable. New -D nbdkit.tls.log and -D nbdkit.tls.session debug flags which enable additional debugging for TLS sessions. The server now responds to "NBD_INFO_NAME" (Eric Blake). Options --run and --vsock may now be used together. In --run, $nbd is now a synonym for $uri instead of having confusing magical properties. Plugins New nbdkit-S3-plugin(1) to access disks stored on AWS S3 and Ceph. New nbdkit-sparse-random-plugin(1) which generates large disks sparsely filled with random blocks of data. These look like virtual disk images and can be used to benchmark copying tools. nbdkit-data-plugin(1) has been greatly enhanced. New features include: "(...)[N:M]" (slice syntax), assignments, comments, variables, "<(SCRIPT)" (inlined shell script). In addition a parse tree is built, allowing optimizations and other enhancements. The plugin is now 64 bit clean allowing patterns larger than 2^32 bytes to work correctly. nbdkit-random-plugin(1) can now be used to test copying tools. nbdkit-vddk-plugin(1) supports enabling compression between VDDK and the VMware server which in some situations can improve performance. Filters New nbdkit-exportname-filter(1) can be used to extend, mask and modify the exports exposed by a plugin (Eric Blake). New nbdkit-exitwhen-filter(1) allowing the server to gracefully exit on various events including external scripts. New nbdkit-checkwrite-filter(1) can be used to benchmark copying tools. nbdkit-ip-filter(1) now supports filtering clients by PID, user ID or group ID, for clients connected over a Unix domain socket. Also you can filter clients using "AF_VSOCK" and vsock port number. nbdkit-log-filter(1) now logs "get_ready", "after_fork" and "preconnect" methods, and there is a new "logscript" feature allowing scripts to be run on logging events. nbdkit-ext2-filter(1) new option "ext2file=exportname" can be used to allow the client to select which file to see from the filesystem using the NBD export name (Eric Blake). Language bindings Python style cleanups (Eric Blake). The OCaml plugin now correctly acquires and releases the GC lock on all paths in and out. There is a new OCaml "NBDKit.version" function returning the version of nbdkit that the plugin was compiled against. Multiple enhancements to the Rust plugin (Alan Somers). API New ".list_exports", ".default_export" and ".export_description" plugin methods allowing plugins to be queried about what export names they support, and give additional descriptions of exports. New "nbdkit_use_default_export" helper function (Eric Blake). New "nbdkit_strdup_intern", "nbdkit_strndup_intern" "nbdkit_printf_intern" and "nbdkit_vprintf_intern" functions which allow you to allocate a string in a plugin without worrying about freeing it (Eric Blake). New "nbdkit_peer_pid", "nbdkit_peer_uid" and "nbdkit_peer_gid" allows plugins and filters to read the client’s credentials, for clients connected over a Unix domain socket. Tests Many tests now use libnbd(3), nbdsh(1), nbdinfo(1) and nbdcopy(1) as lighter weight replacements for qemu tools. Fix compatibility with qemu > 5.2.0. Build Downstream packagers can now use this to add extra downstream information which is printed in version strings etc: ./configure --with-extra=... If ocamldoc(1) is installed at build time, then it will be used to build the NBDKit(3) manual page for OCaml documentation. SEE ALSO nbdkit(1). AUTHORS Authors of nbdkit 1.24: • Alan Somers • Eric Blake • Richard W.M. Jones COPYRIGHT Copyright (C) 2020-2021 Red Hat Inc. LICENSE Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. • Neither the name of Red Hat nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. nbdkit-1.24.0 2021-01-07 nbdkit-release-notes-1.24(1) ------- These are the release notes for libnbd stable release 1.6. This describes the major changes since 1.4. libnbd 1.6.0 was released on 7 January 2021. Security There were no security issues found in this release. If you find a security issue, please read SECURITY in the source (online here: https://github.com/libguestfs/libnbd/blob/master/SECURITY). To find out about previous security issues in libnbd, see libnbd-security(3). New APIs nbd_set_strict_mode(3) nbd_get_strict_mode(3) This API allows you to control how libnbd handles requests from the program that it thinks would trigger undefined behaviour in the server (such as sending unknown commands, zero-sized or out-of- bounds requests, etc). Normally libnbd is very strict and will prevent you from sending such requests, but you can use this API to relax these restrictions, leaving it up to the server to handle or reject them. Some servers may crash if you do this. nbd_get_nr_meta_contexts(3) nbd_get_meta_context(3) nbd_clear_meta_contexts(3) These APIs allow you to review or clear the requested metadata contexts. nbd_opt_list_meta_context(3) nbd_aio_opt_list_meta_context(3) Request that the server lists available metadata contexts. You can only use this in opt mode. Thanks: Eric Blake for all of the new APIs. Enhancements to existing APIs nbd_shutdown(3) new flag "LIBNBD_SHUTDOWN_ABANDON_PENDING" drops any commands in flight which have not been sent to the server yet (Eric Blake). All APIs which have flag (bitmask) parameters now define a "LIBNBD_*_MASK" constant allowing you to tell which bits represent valid flags at compilation time (Eric Blake). New features The performance of the nbdcopy(1) tool has been greatly improved. Also it supports numerous new features. New nbdinfo(1) --map option showing the allocated and sparse extents of an NBD server. This can also show other information such as dirty bitmaps. Tools nbdfuse(1) can now mount the NBD export on top of any regular file, it does not have to mount over a directory. nbdsh(1) now produces a nice error message if the URI in the -u option or the command in the -c option is wrong (Eric Blake). New nbdsh(1) --opt-mode option which requests opt mode on the handle before connecting. nbdinfo(1) --list option now shows the supported metadata contexts of the server (Eric Blake). The output of nbdinfo(1) is now usually printed atomically, ie. it will either print the complete output or an error message, but not partial output followed by an error. nbdinfo(1) can now decode "qemu:allocation-depth" metacontext (Eric Blake). Other improvements and bug fixes Multiple improvements to handling of NBD protocol (Eric Blake). Fix handling of booleans in Python bindings (Eric Blake). Fix handling of flags parameters in OCaml bindings (Eric Blake). Python style fixes (Eric Blake). Fix multiple memory leaks (Eric Blake). OCaml bindings now include ocamldoc-generated documentation, if ocamldoc is installed at build time. The "vector" library from nbdkit is now used by libnbd, simplifying places where we used to use realloc(3). SEE ALSO libnbd(3). AUTHORS Eric Blake Richard W.M. Jones COPYRIGHT Copyright (C) 2020-2021 Red Hat Inc. LICENSE This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA libnbd-1.6.0 2021-01-07 libnbd-release-notes-1.6(1) -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW