Using libblkio (https://libblkio.gitlab.io/) this implements a plugin for reading and writing various disk sources mainly used in virtualization and other high performance cases. These include: NVMe, vhost-user, vhost-vdpa and VFIO PCI. Note the current implementation uses the simplest libblkio API mode (blocking) and so it is not suitable for high performance access. --- plugins/blkio/nbdkit-blkio-plugin.pod | 117 +++++++++++ configure.ac | 20 ++ plugins/blkio/Makefile.am | 80 ++++++++ plugins/blkio/blkio.c | 270 ++++++++++++++++++++++++++ README.md | 4 + TODO | 7 + 6 files changed, 498 insertions(+) diff --git a/plugins/blkio/nbdkit-blkio-plugin.pod b/plugins/blkio/nbdkit-blkio-plugin.pod new file mode 100644 index 000000000..c76784687 --- /dev/null +++ b/plugins/blkio/nbdkit-blkio-plugin.pod @@ -0,0 +1,117 @@ +=head1 NAME + +nbdkit-blkio-plugin - libblkio plugin for NVMe, vhost-user, vDPA, VFIO + +=head1 SYNOPSIS + + nbdkit blkio [driver=]DRIVER [path=FILENAME] [num-queues=N] ... + +=head1 DESCRIPTION + +nbdkit-blkio-plugin is an L<nbdkit(1)> plugin for using +L<libblkio|https://libblkio.gitlab.io/> to access various disk sources +used for high performance applications and virtualization. These +include: NVMe, vhost-user, vDPA and VFIO. + +The first parameter after the plugin name should be the L<libblkio +driver|https://libblkio.gitlab.io/libblkio/blkio.html#drivers>. For +example: + + nbdkit blkio virtio-blk-vhost-user path=vhost.sock + +=head2 Driver: C<nvme-io_uring> + + nbdkit blkio nvme-io_uring path=/dev/ng0n1 + +Connect to an NVMe device, issuing commands through Linux io_uring +(requires Linux E<ge> 5.19). + +=head2 Driver: C<virtio-blk-vfio-pci> + + nbdkit blkio virtio-blk-vfio-pci path=/sys/bus/pci/devices/0000:00:01.0 + +Connect to a PCI device which implements virtio-blk using VFIO. The +path is the path to the device's sysfs directory (see L<lspci(8)>). + +=head2 Driver: C<virtio-blk-vhost-user> + + nbdkit blkio virtio-blk-vhost-user path=vhost.sock + +Connect to a vhost-user-blk device, such as one exported by +L<qemu-storage-daemon(1)>. The path is the vhost-user Unix domain +socket. + +=head2 Driver: C<virtio-blk-vhost-vdpa> + + nbdkit blkio virtio-blk-vhost-vdpa path=chardev + +Connect to a vDPA device which might be implemented in software +(eg. VDUSE) or hardware. The path is the vhost-vdpa character device. + +=head2 Driver: C<io_uring> + + nbdkit blkio io_uring path=FILENAME + +You can use this driver to access local files and block devices +through the libblkio C<io_uring> driver, but it is usually better and +easier to use L<nbdkit-file-plugin(1)>. + +=head1 PARAMETERS + +=over 4 + +=item [B<driver=>]DRIVER + +The name of the libblkio driver to use. + +This parameter is required. + +C<driver=> is a magic config key and may be omitted in most cases. +See L<nbdkit(1)/Magic parameters>. + +=item PROPERTYB<=>VALUE + +Properties such as C<path>, C<num-entries> etc are translated to +libblkio properties. Consult the L<libblkio +documentation|https://libblkio.gitlab.io/libblkio/blkio.html> for a +complete list. + +=item B<get=>PROPERTY + +Get (print) the value of a property after connecting. The property is +fetched and printed in nbdkit debug output, so you will need to use +the I<--verbose> flag. This is useful for debugging. + +=back + +=head1 FILES + +=over 4 + +=item F<$plugindir/nbdkit-blkio-plugin.so> + +The plugin. + +Use C<nbdkit --dump-config> to find the location of C<$plugindir>. + +=back + +=head1 VERSION + +C<nbdkit-blkio-plugin> first appeared in nbdkit 1.34. + +=head1 SEE ALSO + +L<nbdkit-file-plugin(1)>, +L<lspci(8)>, +L<qemu-storage-daemon(1)>, +L<https://libblkio.gitlab.io>, +L<https://libblkio.gitlab.io/libblkio/blkio.html>. + +=head1 AUTHORS + +Richard W.M. Jones + +=head1 COPYRIGHT + +Copyright (C) 2014-2023 Red Hat Inc. diff --git a/configure.ac b/configure.ac index e03374b12..6afeac38e 100644 --- a/configure.ac +++ b/configure.ac @@ -74,6 +74,7 @@ lang_plugins="\ tcl \ " non_lang_plugins="\ + blkio \ cdi \ curl \ data \ @@ -1094,6 +1095,23 @@ AS_IF([test "x$enable_golang" != "xno"],[ ],[GOLANG=no]) AM_CONDITIONAL([HAVE_GOLANG],[test "x$GOLANG" != "xno"]) +dnl Check for libblkio (only if you want to compile the blkio plugin). +AC_ARG_WITH([libblkio], + [AS_HELP_STRING([--without-libblkio], + [disable blkio plugin @<:@default=check@:>@])], + [], + [with_libblkio=check]) +AS_IF([test "$with_libblkio" != "no"],[ + PKG_CHECK_MODULES([LIBBLKIO], [blkio],[ + printf "libblkio version is "; $PKG_CONFIG --modversion blkio + AC_SUBST([LIBBLKIO_CFLAGS]) + AC_SUBST([LIBBLKIO_LIBS]) + AC_DEFINE([HAVE_LIBBLKIO],[1],[libblkio found at compile time.]) + ], + [AC_MSG_WARN([libblkio not found, blkio plugin will be disabled])]) +]) +AM_CONDITIONAL([HAVE_LIBBLKIO],[test "x$LIBBLKIO_LIBS" != "x"]) + dnl Check for curl (only if you want to compile the curl plugin). AC_ARG_WITH([curl], [AS_HELP_STRING([--without-curl], @@ -1402,6 +1420,7 @@ AC_CONFIG_FILES([Makefile include/Makefile include/nbdkit-version.h plugins/Makefile + plugins/blkio/Makefile plugins/cc/Makefile plugins/cdi/Makefile plugins/curl/Makefile @@ -1533,6 +1552,7 @@ feature "TLS" test "x$GNUTLS_LIBS" != "x" echo echo "Optional plugins:" echo +feature "blkio" test "x$HAVE_LIBBLKIO_TRUE" = "x" feature "curl" test "x$HAVE_CURL_TRUE" = "x" feature "example4" test "x$HAVE_PERL_TRUE" = "x" feature "floppy" test "x$HAVE_ICONV_TRUE" = "x" diff --git a/plugins/blkio/Makefile.am b/plugins/blkio/Makefile.am new file mode 100644 index 000000000..dbedcc2eb --- /dev/null +++ b/plugins/blkio/Makefile.am @@ -0,0 +1,80 @@ +# nbdkit +# Copyright (C) 2014-2023 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +include $(top_srcdir)/common-rules.mk + +EXTRA_DIST = nbdkit-blkio-plugin.pod + +if HAVE_LIBBLKIO + +plugin_LTLIBRARIES = nbdkit-blkio-plugin.la + +nbdkit_blkio_plugin_la_SOURCES = \ + blkio.c \ + $(top_srcdir)/include/nbdkit-plugin.h \ + $(NULL) + +nbdkit_blkio_plugin_la_CPPFLAGS = \ + -I$(top_srcdir)/include \ + -I$(top_builddir)/include \ + -I$(top_srcdir)/common/utils \ + $(NULL) +nbdkit_blkio_plugin_la_CFLAGS = \ + $(WARNINGS_CFLAGS) \ + $(LIBBLKIO_CFLAGS) \ + $(NULL) +nbdkit_blkio_plugin_la_LIBADD = \ + $(top_builddir)/common/utils/libutils.la \ + $(IMPORT_LIBRARY_ON_WINDOWS) \ + $(LIBBLKIO_LIBS) \ + $(NULL) +nbdkit_blkio_plugin_la_LDFLAGS = \ + -module -avoid-version -shared $(NO_UNDEFINED_ON_WINDOWS) \ + $(NULL) +if USE_LINKER_SCRIPT +nbdkit_blkio_plugin_la_LDFLAGS += \ + -Wl,--version-script=$(top_srcdir)/plugins/plugins.syms +endif + +if HAVE_POD + +man_MANS = nbdkit-blkio-plugin.1 +CLEANFILES += $(man_MANS) + +nbdkit-blkio-plugin.1: nbdkit-blkio-plugin.pod \ + $(top_builddir)/podwrapper.pl + $(PODWRAPPER) --section=1 --man $@ \ + --html $(top_builddir)/html/$@.html \ + $< + +endif HAVE_POD + +endif diff --git a/plugins/blkio/blkio.c b/plugins/blkio/blkio.c new file mode 100644 index 000000000..058850f63 --- /dev/null +++ b/plugins/blkio/blkio.c @@ -0,0 +1,270 @@ +/* nbdkit + * Copyright (C) 2014-2023 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <errno.h> + +#include <blkio.h> + +#define NBDKIT_API_VERSION 2 +#include <nbdkit-plugin.h> + +#include "vector.h" +#include "const-string-vector.h" + +/* libblkio could do parallel, but we would need to reimplement this + * plugin to use the libblkio event model. + */ +#define THREAD_MODEL NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS + +struct property { + const char *name; + const char *value; +}; +DEFINE_VECTOR_TYPE(properties, struct property) + +static const char *driver = NULL; /* driver name - required */ +static properties props = empty_vector; /* other command line params */ +static const_string_vector gets = empty_vector; /* get= parameters */ + +static void +bio_unload (void) +{ + properties_reset (&props); + const_string_vector_reset (&gets); +} + +/* Called for each key=value passed on the command line. */ +static int +bio_config (const char *key, const char *value) +{ + if (strcmp (key, "driver") == 0) { + if (driver != NULL) { + nbdkit_error ("'driver' property set more than once"); + return -1; + } + driver = value; + } + else if (strcmp (key, "get") == 0) { + if (const_string_vector_append (&gets, value) == -1) + return -1; + } + else if (strcmp (key, "read-only") == 0) { + nbdkit_error ("do not set the libblkio \"read-only\" parameter, " + "use the nbdkit -r flag if read-only is required"); + return -1; + } + else /* general property */ { + struct property prop = { .name = key, .value = value }; + if (properties_append (&props, prop) == -1) + return -1; + } + + return 0; +} + +/* Check the user did pass a driver parameter. */ +static int +bio_config_complete (void) +{ + if (driver == NULL) { + nbdkit_error ("you must supply the driver=<DRIVER> parameter " + "after the plugin name on the command line"); + return -1; + } + + return 0; +} + +#define bio_config_help \ + "driver=<DRIVER> (required) Driver name (eg. \"nvme-io_uring\").\n" \ + "PROPERTY=VALUE Set arbitrary libblkio property.\n" \ + "get=PROPERTY Print property name after connection." + +/* XXX Should be possible to query this from libblkio. */ +static bool +is_preconnect_property (const char *name) +{ + return strcmp (name, "can-add-queues") == 0 || + strcmp (name, "driver") == 0 || + strcmp (name, "fd") == 0 || + strcmp (name, "path") == 0 || + strcmp (name, "read-only") == 0; +} + +/* Create the per-connection handle. */ +static void * +bio_open (int readonly) +{ + struct blkio *b; + size_t i; + + errno = blkio_create (driver, &b); + if (errno != 0) { + nbdkit_error ("blkio_create: error opening driver: %s: %m", driver); + return NULL; + } + + /* Always set the read-only property to a true or false value. */ + errno = blkio_set_bool (b, "read-only", !!readonly); + if (errno != 0) { + nbdkit_error ("error setting property: read-only=%s: %m", + readonly ? "true" : "false"); + blkio_destroy (&b); + return NULL; + } + + /* Set the pre-connect properties. */ + for (i = 0; i < props.len; ++i) { + const struct property *prop = &props.ptr[i]; + + if (is_preconnect_property (prop->name)) { + errno = blkio_set_str (b, prop->name, prop->value); + if (errno != 0) { + nbdkit_error ("error setting property: %s=%s: %m", + prop->name, prop->value); + blkio_destroy (&b); + return NULL; + } + } + } + + /* Connect. */ + errno = blkio_connect (b); + if (errno != 0) { + nbdkit_error ("blkio_connect: failed to connect to device: %m"); + blkio_destroy (&b); + return NULL; + } + + /* Set the post-connect properties. */ + for (i = 0; i < props.len; ++i) { + const struct property *prop = &props.ptr[i]; + + if (! is_preconnect_property (prop->name)) { + errno = blkio_set_str (b, prop->name, prop->value); + if (errno != 0) { + nbdkit_error ("error setting property: %s=%s: %m", + prop->name, prop->value); + blkio_destroy (&b); + return NULL; + } + } + } + + /* Start the block device. */ + errno = blkio_start (b); + if (errno != 0) { + nbdkit_error ("blkio_start: failed to start device: %m"); + blkio_destroy (&b); + return NULL; + } + + /* Print any properties requested on the command line. */ + for (i = 0; i < gets.len; ++i) { + const char *name = gets.ptr[i]; + char *value = NULL; + + if (blkio_get_str (b, name, &value) == 0) + nbdkit_debug ("get %s = %s", name, value); + else + nbdkit_debug ("could not get property %s: %m", name); + free (value); + } + + return b; +} + +/* Close the handle. */ +static void +bio_close (void *handle) +{ + struct blkio *b = handle; + + blkio_destroy (&b); +} + +/* Get the device size. */ +static int64_t +bio_get_size (void *handle) +{ + struct blkio *b = handle; + uint64_t r; + + errno = blkio_get_uint64 (b, "capacity", &r); + if (errno != 0) { + nbdkit_error ("error reading device capacity: %m"); + return -1; + } + + return r; +} + +/* Read data from the device. */ +static int +bio_pread (void *handle, void *buf, uint32_t count, uint64_t offset, + uint32_t flags) +{ + nbdkit_error ("XXX NOT IMPL XXX"); + return -1; +} + +/* Write data to the device. */ +static int +bio_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset, + uint32_t flags) +{ + nbdkit_error ("XXX NOT IMPL XXX"); + return -1; +} + +static struct nbdkit_plugin plugin = { + .name = "blkio", + .version = PACKAGE_VERSION, + .unload = bio_unload, + .config = bio_config, + .config_complete = bio_config_complete, + .config_help = bio_config_help, + .magic_config_key = "driver", + .open = bio_open, + .close = bio_close, + .get_size = bio_get_size, + .pread = bio_pread, + .pwrite = bio_pwrite, + .errno_is_preserved = 1, +}; + +NBDKIT_REGISTER_PLUGIN(plugin) diff --git a/README.md b/README.md index eff94379b..a204ccf39 100644 --- a/README.md +++ b/README.md @@ -118,6 +118,10 @@ For the bittorrent plugin: * [libtorrent-rasterbar](https://www.libtorrent.org) +For the blkio plugin: + +* [libblkio](https://libblkio.gitlab.io/) + For the containerized data importer (CDI) plugin: * podman diff --git a/TODO b/TODO index feeb09289..d4c85df9b 100644 --- a/TODO +++ b/TODO @@ -155,6 +155,13 @@ nbdkit-cdi-plugin: * Look at using skopeo instead of podman pull (https://github.com/containers/skopeo) +nbdkit-blkio-plugin: + +* Use event-driven mode instead of blocking mode. This involves + restructuring the plugin so that there is one or more background + threads to handle the events, and nbdkit threads issue requests to + these threads. (See how it is done in the VDDK plugin.) + Suggestions for language plugins -------------------------------- -- 2.37.3