[PATCH nbdkit v2] Add the ability to write plugins in golang.
Thanks: Dan Berrangé XXX UNFINISHED: - Is using uintptr for the handle a good idea? Plugins must return something != 0. In other languages we would allow plugins to return an arbitrary object here, but this is not possible in golang because of lack of GC roots. - Default can_* methods are hard to implement. Ideally we would be able to test if a user plugin implements a particular callback (eg. TestPlugin.PWrite) or else defaults to the default callback, but even with reflection I don't think this is possible in golang. - Write wrappers etc for all the other methods. - Write documentation. --- plugins/golang/nbdkit-golang-plugin.pod | 34 +++ configure.ac | 32 +++ plugins/golang/Makefile.am | 80 +++++++ .../src/libguestfs.org/nbdkit/wrappers.h | 39 +++ plugins/golang/config-test.go | 38 +++ .../src/libguestfs.org/nbdkit/nbdkit.go | 224 ++++++++++++++++++ .../golang/src/libguestfs.org/nbdkit/utils.go | 75 ++++++ .../src/libguestfs.org/nbdkit/wrappers.go | 88 +++++++ plugins/golang/test/run-test.sh | 42 ++++ plugins/golang/test/test.go | 102 ++++++++ .gitignore | 2 + README | 4 + 12 files changed, 760 insertions(+) diff --git a/plugins/golang/nbdkit-golang-plugin.pod b/plugins/golang/nbdkit-golang-plugin.pod new file mode 100644 index 00000000..b449a830 --- /dev/null +++ b/plugins/golang/nbdkit-golang-plugin.pod @@ -0,0 +1,34 @@ +=head1 NAME + +nbdkit-golang-plugin - writing nbdkit plugins in Go + +=head1 SYNOPSIS + + nbdkit /path/to/plugin.so [arguments...] + +=head1 DESCRIPTION + +This manual page describes how to write nbdkit plugins in compiled +Golang code. Go plugins are compiled to F<*.so> files (the same as +plugins written in C) and are used in the same way. + +XXX MORE DOCS + + + +=head1 VERSION + +Golang plugins first appeared in nbdkit 1.20. + +=head1 SEE ALSO + +L<nbdkit(1)>, +L<nbdkit-plugin(3)>. + +=head1 AUTHORS + +Richard W.M. Jones + +=head1 COPYRIGHT + +Copyright (C) 2020 Red Hat Inc. diff --git a/configure.ac b/configure.ac index c1aec8fa..7fbf7abe 100644 --- a/configure.ac +++ b/configure.ac @@ -49,6 +49,7 @@ LT_INIT dnl List of plugins and filters. lang_plugins="\ + golang \ lua \ ocaml \ perl \ @@ -754,6 +755,34 @@ AS_IF([test "x$enable_lua" != "xno"],[ ]) AM_CONDITIONAL([HAVE_LUA],[test "x$enable_lua" = "xyes"]) +dnl Check for golang. +AC_ARG_ENABLE([golang], + AS_HELP_STRING([--disable-golang], [disable Go language plugin]), + [], + [enable_golang=yes]) +AS_IF([test "x$enable_golang" != "xno"],[ + AC_CHECK_PROG([GOLANG],[go],[go],[no]) + AS_IF([test "x$GOLANG" != "xno"],[ + AC_MSG_CHECKING([if $GOLANG is usable]) + AS_IF([$GOLANG run $srcdir/plugins/golang/config-test.go 2>&AS_MESSAGE_LOG_FD],[ + AC_MSG_RESULT([yes]) + + # Substitute some golang environment. + GOOS=`$GOLANG env GOOS` + GOARCH=`$GOLANG env GOARCH` + GOROOT=`$GOLANG env GOROOT` + AC_SUBST([GOOS]) + AC_SUBST([GOARCH]) + AC_SUBST([GOROOT]) + ],[ + AC_MSG_RESULT([no]) + AC_MSG_WARN([golang ($GOLANG) is installed but not usable]) + GOLANG=no + ]) + ]) +],[GOLANG=no]) +AM_CONDITIONAL([HAVE_GOLANG],[test "x$GOLANG" != "xno"]) + dnl Check for curl (only if you want to compile the curl plugin). AC_ARG_WITH([curl], [AS_HELP_STRING([--without-curl], @@ -1002,6 +1031,7 @@ AC_CONFIG_FILES([Makefile plugins/file/Makefile plugins/floppy/Makefile plugins/full/Makefile + plugins/golang/Makefile plugins/guestfs/Makefile plugins/gzip/Makefile plugins/info/Makefile @@ -1125,6 +1155,8 @@ feature "vddk ................................... " \ echo echo "Languages:" echo +feature "go ..................................... " \ + test "x$HAVE_GOLANG_TRUE" = "x" feature "lua .................................... " \ test "x$HAVE_LUA_TRUE" = "x" feature "ocaml .................................. " \ diff --git a/plugins/golang/Makefile.am b/plugins/golang/Makefile.am new file mode 100644 index 00000000..4f69f03f --- /dev/null +++ b/plugins/golang/Makefile.am @@ -0,0 +1,80 @@ +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +include $(top_srcdir)/common-rules.mk + +EXTRA_DIST = \ + nbdkit-golang-plugin.pod \ + src/libguestfs.org/nbdkit/nbdkit.go \ + src/libguestfs.org/nbdkit/utils.go \ + src/libguestfs.org/nbdkit/wrappers.go \ + src/libguestfs.org/nbdkit/wrappers.h \ + test/run-test.sh \ + test/test.go \ + $(NULL) + +if HAVE_GOLANG + +# There is nothing to build. Everything is statically compiled and +# linked together when we compile the test. + +TESTS = test/run-test.sh +check_DATA = test/nbdkit-gotest-plugin.so + +test/nbdkit-gotest-plugin.so: \ + src/libguestfs.org/nbdkit/nbdkit.go \ + src/libguestfs.org/nbdkit/utils.go \ + src/libguestfs.org/nbdkit/wrappers.go \ + src/libguestfs.org/nbdkit/wrappers.h \ + test/test.go + cd test && \ + GOPATH="$(abs_builddir)" \ + $(GOLANG) build \ + -o nbdkit-gotest-plugin.so -buildmode=c-shared + +CLEANFILES += \ + test/nbdkit-gotest-plugin.h \ + test/nbdkit-gotest-plugin.so \ + $(NULL) + +if HAVE_POD + +man_MANS = nbdkit-golang-plugin.3 +CLEANFILES += $(man_MANS) + +nbdkit-golang-plugin.3: nbdkit-golang-plugin.pod + $(PODWRAPPER) --section=3 --man $@ \ + --html $(top_builddir)/html/$@.html \ + $< + +endif HAVE_POD + +endif HAVE_GOLANG diff --git a/plugins/golang/src/libguestfs.org/nbdkit/wrappers.h b/plugins/golang/src/libguestfs.org/nbdkit/wrappers.h new file mode 100644 index 00000000..e385663f --- /dev/null +++ b/plugins/golang/src/libguestfs.org/nbdkit/wrappers.h @@ -0,0 +1,39 @@ +/* cgo wrappers. + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +extern int config_wrapper (const char *key, const char *value); +extern int config_complete_wrapper (void); +extern void *open_wrapper (int readonly); +extern void close_wrapper (void *handle); +extern int64_t get_size_wrapper (void *handle); +extern int pread_wrapper (void *handle, void *buf, + uint32_t count, uint64_t offset, uint32_t flags); diff --git a/plugins/golang/config-test.go b/plugins/golang/config-test.go new file mode 100644 index 00000000..0f5cfe6b --- /dev/null +++ b/plugins/golang/config-test.go @@ -0,0 +1,38 @@ +/* Go configuration test + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* Used by ./configure to check golang is functional. */ + +package main + +func main() { +} diff --git a/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go b/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go new file mode 100644 index 00000000..3abf39bc --- /dev/null +++ b/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go @@ -0,0 +1,224 @@ +/* Go helper functions. + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +package nbdkit + +/* +#cgo pkg-config: nbdkit +#cgo LDFLAGS: -Wl,--unresolved-symbols=ignore-in-object-files + +#include <stdio.h> +#include <stdlib.h> + +#define NBDKIT_API_VERSION 2 +#include <nbdkit-plugin.h> +#include "wrappers.h" +*/ +import "C" + +import ( + "fmt" + "syscall" + "unsafe" +) + +// The plugin may raise errors by returning this struct (instead of nil). +type PluginError struct { + Errmsg string // string (passed to nbdkit_error) + Errno syscall.Errno // errno (optional, use 0 if not available) +} + +func (e PluginError) String() string { + if e.Errno != 0 { + return e.Errmsg + } else { + return fmt.Sprintf("%s (errno %d)", e.Errmsg, e.Errno) + } +} + +func (e PluginError) Error() string { + return e.String() +} + +// The plugin interface. +type PluginInterface interface { + // Open, GetSize and PRead are required for all plugins. + // Other methods are optional. + Config(key string, value string) error + ConfigComplete() error + Open(readonly bool) (uintptr, error) + Close(handle uintptr) + GetSize(handle uintptr) (uint64, error) + PRead(handle uintptr, buf []byte, offset uint64, + flags uint32) error +} + +// Default implementations for plugin interface methods. +type Plugin struct{} + +func (p Plugin) Config(key string, value string) error { + return nil +} + +func (p Plugin) ConfigComplete() error { + return nil +} + +func (p Plugin) Open(readonly bool) (uintptr, error) { + panic("plugin must implement Open()") +} + +func (p Plugin) Close(hande uintptr) { +} + +func (p Plugin) GetSize(handle uintptr) (uint64, error) { + panic("plugin must implement GetSize()") +} + +func (p Plugin) PRead(handle uintptr, buf []byte, + offset uint64, flags uint32) error { + panic("plugin must implement PRead()") +} + +// The implementation of the user plugin. +var pluginImpl PluginInterface + +// Callbacks from the server. These translate C to Go and back. + +func set_error(err error) { + perr, ok := err.(PluginError) + if ok { + if perr.Errno != 0 { + SetError(perr.Errno) + } + Error(perr.Errmsg) + } else { + Error(err.Error()) + } +} + +//export implConfig +func implConfig(key *C.char, value *C.char) C.int { + err := pluginImpl.Config(C.GoString(key), C.GoString(value)) + if err != nil { + set_error(err) + return -1 + } + return 0 +} + +//export implConfigComplete +func implConfigComplete() C.int { + err := pluginImpl.ConfigComplete() + if err != nil { + set_error(err) + return -1 + } + return 0 +} + +//export implOpen +func implOpen(c_readonly C.int) unsafe.Pointer { + readonly := false + if c_readonly != 0 { + readonly = true + } + h, err := pluginImpl.Open(readonly) + if err != nil { + set_error(err) + return nil + } + if h == 0 { + panic("Open method: handle must be != 0") + } + return unsafe.Pointer(h) +} + +//export implClose +func implClose(handle unsafe.Pointer) { + pluginImpl.Close(uintptr(handle)) +} + +//export implGetSize +func implGetSize(handle unsafe.Pointer) C.int64_t { + size, err := pluginImpl.GetSize(uintptr(handle)) + if err != nil { + set_error(err) + return -1 + } + return C.int64_t(size) +} + +//export implPRead +func implPRead(handle unsafe.Pointer, buf unsafe.Pointer, + count C.uint32_t, offset C.uint64_t, flags C.uint32_t) C.int { + err := pluginImpl.PRead(uintptr(handle), + C.GoBytes(buf, C.int(count)), + uint64(offset), uint32(flags)) + if err != nil { + set_error(err) + return -1 + } + return 0 +} + +// Called from C plugin_init function. +func PluginInitialize(name string, impl PluginInterface) unsafe.Pointer { + pluginImpl = impl + + plugin := C.struct_nbdkit_plugin{} + + // Set up the hidden plugin fields as for C. + struct_size := C.ulong(unsafe.Sizeof(plugin)) + plugin._struct_size = struct_size + plugin._api_version = C.NBDKIT_API_VERSION + plugin._thread_model = C.NBDKIT_THREAD_MODEL_PARALLEL + + // Set up the other fields. + plugin.name = C.CString(name) + plugin.config = (*[0]byte)(C.config_wrapper) + plugin.config_complete = (*[0]byte)(C.config_complete_wrapper) + plugin.open = (*[0]byte)(C.open_wrapper) + plugin.close = (*[0]byte)(C.close_wrapper) + plugin.get_size = (*[0]byte)(C.get_size_wrapper) + plugin.pread = (*[0]byte)(C.pread_wrapper) + + // Golang plugins don't preserve errno correctly. + plugin.errno_is_preserved = 0 + + // Return a newly malloced copy of the struct. This must be + // globally available to the C code in the server, so it is + // never freed. + p := (*C.struct_nbdkit_plugin)(C.malloc(struct_size)) + *p = plugin + return unsafe.Pointer(p) +} diff --git a/plugins/golang/src/libguestfs.org/nbdkit/utils.go b/plugins/golang/src/libguestfs.org/nbdkit/utils.go new file mode 100644 index 00000000..d9c0c188 --- /dev/null +++ b/plugins/golang/src/libguestfs.org/nbdkit/utils.go @@ -0,0 +1,75 @@ +/* cgo wrappers. + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +package nbdkit + +/* +#cgo pkg-config: nbdkit + +#define NBDKIT_API_VERSION 2 +#include <nbdkit-plugin.h> + +// cgo cannot call varargs functions. +void +_nbdkit_debug (const char *s) +{ + nbdkit_debug ("%s", s); +} + +// cgo cannot call varargs functions. +void +_nbdkit_error (const char *s) +{ + nbdkit_error ("%s", s); +} +*/ +import "C" +import "syscall" + +// Utility functions. + +func Debug(s string) { + C._nbdkit_debug(C.CString(s)) +} + +// This function is provided but plugins would rarely need to call +// this explicitly since returning an error from a plugin callback +// will call it implicitly. +func Error(s string) { + C._nbdkit_error(C.CString(s)) +} + +// Same applies as for Error(). Callers should not usually need to +// call this. +func SetError(err syscall.Errno) { + C.nbdkit_set_error(C.int(err)) +} diff --git a/plugins/golang/src/libguestfs.org/nbdkit/wrappers.go b/plugins/golang/src/libguestfs.org/nbdkit/wrappers.go new file mode 100644 index 00000000..b18521f1 --- /dev/null +++ b/plugins/golang/src/libguestfs.org/nbdkit/wrappers.go @@ -0,0 +1,88 @@ +/* cgo wrappers. + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +package nbdkit + +/* +#cgo pkg-config: nbdkit + +#include <stdio.h> +#include <stdlib.h> + +#define NBDKIT_API_VERSION 2 +#include <nbdkit-plugin.h> +#include "wrappers.h" + +extern int implConfig (); +int +config_wrapper (const char *key, const char *value) +{ + return implConfig (key, value); +} + +extern int implConfigComplete (); +int +config_complete_wrapper (void) +{ + return implConfigComplete (); +} + +extern void *implOpen (); +void * +open_wrapper (int readonly) +{ + return implOpen (readonly); +} + +extern void implClose (); +void +close_wrapper (void *handle) +{ + return implClose (handle); +} + +extern int64_t implGetSize (); +int64_t +get_size_wrapper (void *handle) +{ + return implGetSize (handle); +} + +extern int implPRead (); +int +pread_wrapper (void *handle, void *buf, + uint32_t count, uint64_t offset, uint32_t flags) +{ + return implPRead (handle, buf, count, offset, flags); +} +*/ +import "C" diff --git a/plugins/golang/test/run-test.sh b/plugins/golang/test/run-test.sh new file mode 100755 index 00000000..f4da139e --- /dev/null +++ b/plugins/golang/test/run-test.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# nbdkit +# Copyright (C) 2020 Red Hat Inc. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# * Neither the name of Red Hat nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF +# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. + +set -e +set -x + +if ! qemu-img --version; then + echo "qemu-img is required to run this test" + exit 1 +fi + +../../nbdkit -f -v test/nbdkit-gotest-plugin.so size=$((1024 * 1024)) \ + --run 'qemu-img info $nbd' diff --git a/plugins/golang/test/test.go b/plugins/golang/test/test.go new file mode 100644 index 00000000..f5b1a33b --- /dev/null +++ b/plugins/golang/test/test.go @@ -0,0 +1,102 @@ +/* Test plugin. + * Copyright (C) 2013-2020 Red Hat Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +package main + +import ( + "C" + "libguestfs.org/nbdkit" + "strconv" + "unsafe" +) + +type TestPlugin struct { + nbdkit.Plugin +} + +var pluginName = "test" +var size uint64 +var size_set = false + +func (p TestPlugin) Config(key string, value string) error { + if key == "size" { + var err error + size, err = strconv.ParseUint(value, 0, 64) + if err != nil { + return err + } + size_set = true + return nil + } else { + return nbdkit.PluginError{Errmsg: "unknown parameter"} + } +} + +func (p TestPlugin) ConfigComplete() error { + if !size_set { + return nbdkit.PluginError{Errmsg: "size parameter is required"} + } + return nil +} + +func (p TestPlugin) Open(readonly bool) (uintptr, error) { + nbdkit.Debug("golang code running in the .open callback") + return 1, nil +} + +func (p TestPlugin) GetSize(handle uintptr) (uint64, error) { + nbdkit.Debug("golang code running in the .get_size callback") + return size, nil +} + +func (p TestPlugin) PRead(handle uintptr, buf []byte, + offset uint64, flags uint32) error { + nbdkit.Debug("golang code running in the .pread callback") + for i := 0; i < len(buf); i++ { + buf[i] = 0 + } + return nil +} + +// This is never(?) called, but must exist. +func main() {} + +//---------------------------------------------------------------------- +// The boilerplate below this line is required by all golang plugins. + +//export plugin_init +func plugin_init() unsafe.Pointer { + // If your plugin needs to do any initialization, put it here. + //... + // Then you must call the following function. + return nbdkit.PluginInitialize(pluginName, &TestPlugin{}) +} diff --git a/.gitignore b/.gitignore index 11cd976b..f681f526 100644 --- a/.gitignore +++ b/.gitignore @@ -72,6 +72,8 @@ plugins/*/*.3 /plugins/example4/nbdkit-example4-plugin /plugins/eval/call.c /plugins/eval/methods.c +/plugins/golang/test/nbdkit-gotest-plugin.h +/plugins/golang/test/nbdkit-gotest-plugin.so /plugins/ocaml/nbdkit-ocamlexample-plugin.so /plugins/rust/Cargo.lock /plugins/rust/Cargo.toml diff --git a/README b/README index a33c5693..62fc93e9 100644 --- a/README +++ b/README @@ -148,6 +148,10 @@ For the Rust plugin: - cargo (other dependencies will be downloaded at build time) +To be able to write plugins in golang: + + - go >= 1.5 + For bash tab completion: - bash-completion >= 1.99 -- 2.25.0
On Tue, Apr 21, 2020 at 11:44:59AM +0100, Richard W.M. Jones wrote:
Thanks: Dan Berrangé
XXX UNFINISHED:
- Is using uintptr for the handle a good idea? Plugins must return something != 0. In other languages we would allow plugins to return an arbitrary object here, but this is not possible in golang because of lack of GC roots.
Yeah, this is the bit that looks wierd to me when thinking about this from a Go dev POV. You asked me on IRC whether we should have a separate interface for a connection object, and this makes me think that we should indeed do that. Of course we still have the problem that C code is forbidden to keep a pointer to a Go object across method calls. IIUC, the C layuer in nbdkit treats the "void *" as a completely opaque value, so we don't actually have to store any valid pointer as the handle at all. It can be literally any value that fits in a 32-bit pointer. Thus we can keep a global variable mapping the GO objects to fake C "pointers" So it would look like this: // The plugin interface. type PluginInterface interface { // Open, GetSize and PRead are required for all plugins. // Other methods are optional. Config(key string, value string) error ConfigComplete() error Open(readonly bool) (PluginConnectionInterface, error) } type PluginConnectionInterface interface { // Open, GetSize and PRead are required for all plugins. // Other methods are optional. Close() GetSize() (uint64, error) PRead(buf []byte, offset uint64, flags uint32) error } Now, we need todo some mapping var connectionID uint var connectionMap map[uint]PluginConnectionInterface //export implOpen func implOpen(c_readonly C.int) unsafe.Pointer { readonly := false if c_readonly != 0 { readonly = true } h, err := pluginImpl.Open(readonly) if err != nil { set_error(err) return nil } if h == 0 { panic("Open method: handle must be != 0") } id := connectionID++ connectionMap[id] = h return unsafe.Pointer(uintptr(id)) } func getConn(handle unsafe.Pointer) PluginConnectionInterface { id := uint(uintptr(handle)) conn, ok = connectionMap[id] if !ok { panic("Connection %d was not open", id) } } //export implClose func implClose(handle unsafe.Pointer) { conn := getConn(handle) conn.Close() delete(connectionMap, id) } //export implGetSize func implGetSize(handle unsafe.Pointer) C.int64_t { conn := getConn(handle) size, err := conn.GetSize() if err != nil { set_error(err) return -1 } return C.int64_t(size) }
diff --git a/plugins/golang/test/test.go b/plugins/golang/test/test.go new file mode 100644 index 00000000..f5b1a33b --- /dev/null +++ b/plugins/golang/test/test.go
type TestPlugin struct { nbdkit.Plugin } type TestPluginConnection struct { nbdkit.PluginConnection ...other per-connection fields... } var pluginName = "test" var size uint64 var size_set = false func (p TestPlugin) Config(key string, value string) error { if key == "size" { var err error size, err = strconv.ParseUint(value, 0, 64) if err != nil { return err } size_set = true return nil } else { return nbdkit.PluginError{Errmsg: "unknown parameter"} } } func (p TestPlugin) ConfigComplete() error { if !size_set {
+ return nbdkit.PluginError{Errmsg: "size parameter is required"} } return nil }
func (p TestPlugin) Open(readonly bool) (nbdkit.PluginConnectionInterface, error) { nbdkit.Debug("golang code running in the .open callback") return &TestPluginConnection{}, nil } func (p TestPluginConnection) GetSize() (uint64, error) { nbdkit.Debug("golang code running in the .get_size callback") return size, nil } func (p TestPluginConnection) PRead(buf []byte, offset uint64, flags uint32) error { nbdkit.Debug("golang code running in the .pread callback") for i := 0; i < len(buf); i++ { buf[i] = 0 } return nil } Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Apr 21, 2020 at 05:07:43PM +0100, Daniel P. Berrangé wrote:
On Tue, Apr 21, 2020 at 11:44:59AM +0100, Richard W.M. Jones wrote:
Thanks: Dan Berrangé
XXX UNFINISHED:
- Is using uintptr for the handle a good idea? Plugins must return something != 0. In other languages we would allow plugins to return an arbitrary object here, but this is not possible in golang because of lack of GC roots.
Yeah, this is the bit that looks wierd to me when thinking about this from a Go dev POV. You asked me on IRC whether we should have a separate interface for a connection object, and this makes me think that we should indeed do that. ...
This is what the patch looks like with a separate connection object. It appears to work. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Thu, Apr 23, 2020 at 10:03:45AM +0100, Richard W.M. Jones wrote:
On Tue, Apr 21, 2020 at 05:07:43PM +0100, Daniel P. Berrangé wrote:
On Tue, Apr 21, 2020 at 11:44:59AM +0100, Richard W.M. Jones wrote:
Thanks: Dan Berrangé
XXX UNFINISHED:
- Is using uintptr for the handle a good idea? Plugins must return something != 0. In other languages we would allow plugins to return an arbitrary object here, but this is not possible in golang because of lack of GC roots.
Yeah, this is the bit that looks wierd to me when thinking about this from a Go dev POV. You asked me on IRC whether we should have a separate interface for a connection object, and this makes me think that we should indeed do that. ...
This is what the patch looks like with a separate connection object. It appears to work.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
From a929cff8745f21df913ea3230741dc7f2a1c016e Mon Sep 17 00:00:00 2001 From: "Richard W.M. Jones" <rjones@redhat.com> Date: Thu, 9 Apr 2020 12:45:10 +0100 Subject: [PATCH] Add the ability to write plugins in golang. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit
Thanks: Dan Berrangé
XXX UNFINISHED:
- Default can_* methods are hard to implement. Ideally we would be able to test if a user plugin implements a particular callback (eg. TestPlugin.PWrite) or else defaults to the default callback, but even with reflection I don't think this is possible in golang. [Discussed on IRC: We decided the only option was to go for nbdkit-sh-plugin style can_* methods]
- Write wrappers etc for all the other methods.
- Write documentation. --- plugins/golang/nbdkit-golang-plugin.pod | 34 +++ configure.ac | 32 +++ plugins/golang/Makefile.am | 80 ++++++ .../src/libguestfs.org/nbdkit/wrappers.h | 41 +++ plugins/golang/config-test.go | 38 +++ .../src/libguestfs.org/nbdkit/nbdkit.go | 270 ++++++++++++++++++ .../golang/src/libguestfs.org/nbdkit/utils.go | 75 +++++ .../src/libguestfs.org/nbdkit/wrappers.go | 102 +++++++ plugins/golang/test/run-test.sh | 42 +++ plugins/golang/test/test.go | 119 ++++++++ .gitignore | 2 + README | 4 + 12 files changed, 839 insertions(+) create mode 100644 plugins/golang/nbdkit-golang-plugin.pod create mode 100644 plugins/golang/Makefile.am create mode 100644 plugins/golang/src/libguestfs.org/nbdkit/wrappers.h create mode 100644 plugins/golang/config-test.go create mode 100644 plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go create mode 100644 plugins/golang/src/libguestfs.org/nbdkit/utils.go create mode 100644 plugins/golang/src/libguestfs.org/nbdkit/wrappers.go create mode 100755 plugins/golang/test/run-test.sh create mode 100644 plugins/golang/test/test.go
I think this design with two interfaces looks pretty nice from the Go plugin implementors POV now. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Apr 23, 2020 at 02:49:25PM +0100, Daniel P. Berrangé wrote:
I think this design with two interfaces looks pretty nice from the Go plugin implementors POV now.
Thanks. While implementing PWrite I discovered that C.GoBytes makes a copy (the docs were unclear on this), so it wasn't actually reading and writing from the server at all :-( All fixed now after reading this very long issue: https://github.com/golang/go/issues/13656 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
participants (2)
-
Daniel P. Berrangé -
Richard W.M. Jones