8:51 a.m.
Sorry Dan, but I really do dislike golang with a passion :-)
Here is a patch that allows you to write nbdkit plugins in golang. As
with C, OCaml and Rust, you can write a plugin in Go which compiles
directly to a .so file that can be loaded into golang, so in that
sense it works completely differently from scripting language plugins
like Perl and Python where there's an nbdkit-<lang>-plugin that
intermediates between nbdkit and the user’s code.
With that said, there are many problems. The root cause of most of
them is that you cannot pass Go pointers to C, perhaps because the
golang developers never heard of registering extra GC roots[1], even
though that is common in non-Blub languages like OCaml and Haskell.
This leads to awkward complications nicely summarised in this page:
https://eli.thegreenplace.net/2019/passing-callbacks-and-pointers-to-cgo/
This requires that every callback has a wrapper across 3 files. Oh
and these wrappers cannot be in a library module, they must be copied
into the plugin source code, AND they must be commented out by the
end-user by hand if the callback is not called. (There may be a way
around the latter issues, but I could not work out how.) This means
plugins have tons of duplicated source code.
Other issues that are not related to that one:
- We have to link the nbdkit module with
-Wl,--unresolved-symbols=ignore-in-object-files because of the way
nbdkit plugins have deliberately unresolved symbols. See the
libnbdkit.so proposal on the mailing list for another way to solve
this.
- Be nice if ./configure could check that golang >= 1.5 since that
was the first version that introduced shared libraries.
- Since initialization is not synchronous in golang, you cannot rely
on anything being initialized in the plugin before nbdkit starts
calling in. For this reason I worked around it by having
plugin_init() call a start up function (func init_plugin()) where
all golang initialization must be done. Otherwise:
nbdkit: golang plugin:
dlopen ("plugin.so");
let's start initializing
init = dlsym ("plugin_init");
return &plugin (uninitialized!)
plugin = init ();
plugin.load ()
hey, I'm still initializing!
- Related to the previous point: Be nice to move plugin_init() into
the nbdkit module. However I don't believe this is possible
because this function has to call into the plugin (in main module).
- Tests are a joke at the moment. We would really need a test which
properly exercises threads / parallel client connections, so we can
be sure that the nbdkit thread & golang goroutine models do not
conflict in some way. (I don't think they do, but need to check).
- Current test func pluginPRead() needs to be completed.
- Documentation needs fixing. I didn't want to write too many docs
until I know finally how plugins would work.
Rich.
[1] Reading the proposal here confirms my suspicions, since there is a
much simpler, better and more obvious solution than what is proposed:
https://github.com/golang/proposal/blob/master/design/12416-cgo-pointers.md
8:51 a.m.
Similar to C, OCaml and Rust, this is not a plugin per se. Instead
it's more of a method and set of tests around writing plugins in
golang. They are standalone programs that compile into shared objects
that nbdkit can then load (so there is no "go plugin" between nbdkit
and the user plugin, unlike in scripting languages like Perl).
---
plugins/golang/nbdkit-golang-plugin.pod | 34 ++++++++
configure.ac | 32 +++++++
plugins/golang/Makefile.am | 75 ++++++++++++++++
plugins/golang/test/wrappers.h | 40 +++++++++
plugins/golang/config-test.go | 38 +++++++++
.../src/libguestfs.org/nbdkit/nbdkit.go | 57 +++++++++++++
plugins/golang/test/init.go | 67 +++++++++++++++
plugins/golang/test/run-test.sh | 41 +++++++++
plugins/golang/test/test.go | 82 ++++++++++++++++++
plugins/golang/test/wrappers.go | 85 +++++++++++++++++++
.gitignore | 2 +
README | 4 +
12 files changed, 557 insertions(+)
diff --git a/plugins/golang/nbdkit-golang-plugin.pod
b/plugins/golang/nbdkit-golang-plugin.pod
new file mode 100644
index 00000000..b449a830
--- /dev/null
+++ b/plugins/golang/nbdkit-golang-plugin.pod
@@ -0,0 +1,34 @@
+=head1 NAME
+
+nbdkit-golang-plugin - writing nbdkit plugins in Go
+
+=head1 SYNOPSIS
+
+ nbdkit /path/to/plugin.so [arguments...]
+
+=head1 DESCRIPTION
+
+This manual page describes how to write nbdkit plugins in compiled
+Golang code. Go plugins are compiled to F<*.so> files (the same as
+plugins written in C) and are used in the same way.
+
+XXX MORE DOCS
+
+
+
+=head1 VERSION
+
+Golang plugins first appeared in nbdkit 1.20.
+
+=head1 SEE ALSO
+
+L<nbdkit(1)>,
+L<nbdkit-plugin(3)>.
+
+=head1 AUTHORS
+
+Richard W.M. Jones
+
+=head1 COPYRIGHT
+
+Copyright (C) 2020 Red Hat Inc.
diff --git a/configure.ac b/configure.ac
index c5db962c..4b1c4cd6 100644
--- a/configure.ac
+++ b/configure.ac
@@ -49,6 +49,7 @@ LT_INIT
dnl List of plugins and filters.
lang_plugins="\
+ golang \
lua \
ocaml \
perl \
@@ -754,6 +755,34 @@ AS_IF([test "x$enable_lua" != "xno"],[
])
AM_CONDITIONAL([HAVE_LUA],[test "x$enable_lua" = "xyes"])
+dnl Check for golang.
+AC_ARG_ENABLE([golang],
+ AS_HELP_STRING([--disable-golang], [disable Go language plugin]),
+ [],
+ [enable_golang=yes])
+AS_IF([test "x$enable_golang" != "xno"],[
+ AC_CHECK_PROG([GOLANG],[go],[go],[no])
+ AS_IF([test "x$GOLANG" != "xno"],[
+ AC_MSG_CHECKING([if $GOLANG is usable])
+ AS_IF([$GOLANG run $srcdir/plugins/golang/config-test.go
2>&AS_MESSAGE_LOG_FD],[
+ AC_MSG_RESULT([yes])
+
+ # Substitute some golang environment.
+ GOOS=`$GOLANG env GOOS`
+ GOARCH=`$GOLANG env GOARCH`
+ GOROOT=`$GOLANG env GOROOT`
+ AC_SUBST([GOOS])
+ AC_SUBST([GOARCH])
+ AC_SUBST([GOROOT])
+ ],[
+ AC_MSG_RESULT([no])
+ AC_MSG_WARN([golang ($GOLANG) is installed but not usable])
+ GOLANG=no
+ ])
+ ])
+],[GOLANG=no])
+AM_CONDITIONAL([HAVE_GOLANG],[test "x$GOLANG" != "xno"])
+
dnl Check for curl (only if you want to compile the curl plugin).
AC_ARG_WITH([curl],
[AS_HELP_STRING([--without-curl],
@@ -1002,6 +1031,7 @@ AC_CONFIG_FILES([Makefile
plugins/file/Makefile
plugins/floppy/Makefile
plugins/full/Makefile
+ plugins/golang/Makefile
plugins/guestfs/Makefile
plugins/gzip/Makefile
plugins/info/Makefile
@@ -1125,6 +1155,8 @@ feature "vddk ................................... " \
echo
echo "Languages:"
echo
+feature "go ..................................... " \
+ test "x$HAVE_GOLANG_TRUE" = "x"
feature "lua .................................... " \
test "x$HAVE_LUA_TRUE" = "x"
feature "ocaml .................................. " \
diff --git a/plugins/golang/Makefile.am b/plugins/golang/Makefile.am
new file mode 100644
index 00000000..961495f8
--- /dev/null
+++ b/plugins/golang/Makefile.am
@@ -0,0 +1,75 @@
+# nbdkit
+# Copyright (C) 2020 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+include $(top_srcdir)/common-rules.mk
+
+EXTRA_DIST = \
+ nbdkit-golang-plugin.pod \
+ src/libguestfs.org/nbdkit/nbdkit.go \
+ test/init.go \
+ test/run-test.sh \
+ test/test.go \
+ test/wrappers.go \
+ test/wrappers.h \
+ $(NULL)
+
+if HAVE_GOLANG
+
+# There is nothing to build. Everything is statically compiled and
+# linked together when we compile the test.
+
+TESTS = test/run-test.sh
+check_DATA = test/nbdkit-gotest-plugin.so
+
+test/nbdkit-gotest-plugin.so: src/libguestfs.org/nbdkit/nbdkit.go test/init.go
test/test.go test/wrappers.go test/wrappers.h
+ cd test && \
+ GOPATH="$(abs_builddir)" \
+ $(GOLANG) build \
+ -o nbdkit-gotest-plugin.so -buildmode=c-shared
+
+CLEANFILES += \
+ test/nbdkit-gotest-plugin.so \
+ test/nbdkit-gotest-plugin.h \
+ $(NULL)
+
+if HAVE_POD
+
+man_MANS = nbdkit-golang-plugin.3
+CLEANFILES += $(man_MANS)
+
+nbdkit-golang-plugin.3: nbdkit-golang-plugin.pod
+ $(PODWRAPPER) --section=3 --man $@ \
+ --html $(top_builddir)/html/$@.html \
+ $<
+
+endif HAVE_POD
+
+endif HAVE_GOLANG
diff --git a/plugins/golang/test/wrappers.h b/plugins/golang/test/wrappers.h
new file mode 100644
index 00000000..ea654631
--- /dev/null
+++ b/plugins/golang/test/wrappers.h
@@ -0,0 +1,40 @@
+/* Go wrappers
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+extern void *open_wrapper (int readonly);
+extern void close_wrapper (void *handle);
+extern int64_t get_size_wrapper (void *handle);
+extern int pread_wrapper (void *handle, void *buf,
+ uint32_t count, uint64_t offset, uint32_t flags);
+extern int pwrite_wrapper (void *handle, const void *buf,
+ uint32_t count, uint64_t offset, uint32_t flags);
+// XXX ALL WRAPPERS
diff --git a/plugins/golang/config-test.go b/plugins/golang/config-test.go
new file mode 100644
index 00000000..0f5cfe6b
--- /dev/null
+++ b/plugins/golang/config-test.go
@@ -0,0 +1,38 @@
+/* Go configuration test
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+/* Used by ./configure to check golang is functional. */
+
+package main
+
+func main() {
+}
diff --git a/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go
b/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go
new file mode 100644
index 00000000..c21eabb3
--- /dev/null
+++ b/plugins/golang/src/libguestfs.org/nbdkit/nbdkit.go
@@ -0,0 +1,57 @@
+/* Go helper functions.
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+package nbdkit
+
+/*
+#cgo pkg-config: nbdkit
+#cgo LDFLAGS: -Wl,--unresolved-symbols=ignore-in-object-files
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#define NBDKIT_API_VERSION 2
+#include <nbdkit-plugin.h>
+
+// cgo cannot call varargs functions.
+void
+_nbdkit_debug (const char *s)
+{
+ nbdkit_debug ("%s", s);
+}
+
+*/
+import "C"
+
+func Debug(s string) {
+ C._nbdkit_debug(C.CString(s))
+}
diff --git a/plugins/golang/test/init.go b/plugins/golang/test/init.go
new file mode 100644
index 00000000..bfb02f62
--- /dev/null
+++ b/plugins/golang/test/init.go
@@ -0,0 +1,67 @@
+/* Initialize the plugin.
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+package main
+
+/*
+#cgo pkg-config: nbdkit
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#define NBDKIT_API_VERSION 2
+#include <nbdkit-plugin.h>
+*/
+import "C"
+import "unsafe"
+
+var plugin C.struct_nbdkit_plugin
+
+//export plugin_init
+func plugin_init() *C.struct_nbdkit_plugin {
+ // Initialization is asynchronous, but need it to happen
+ // synchronously, so we need to provide a function that the
+ // user code can call to set plugin fields.
+ init_plugin()
+
+ // Set up the hidden plugin fields as for C.
+ struct_size := C.ulong(unsafe.Sizeof(plugin))
+ plugin._struct_size = struct_size
+ plugin._api_version = C.NBDKIT_API_VERSION
+ plugin._thread_model = C.NBDKIT_THREAD_MODEL_PARALLEL
+ p := (*C.struct_nbdkit_plugin)(C.malloc(struct_size))
+ *p = plugin
+ return p
+}
+
+// This must exist, but it is never called.
+func main() {}
diff --git a/plugins/golang/test/run-test.sh b/plugins/golang/test/run-test.sh
new file mode 100755
index 00000000..5c10f1b7
--- /dev/null
+++ b/plugins/golang/test/run-test.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# nbdkit
+# Copyright (C) 2020 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+set -e
+set -x
+
+if ! qemu-img --version; then
+ echo "qemu-img is required to run this test"
+ exit 1
+fi
+
+../../nbdkit -f -v test/nbdkit-gotest-plugin.so --run 'qemu-img info $nbd'
diff --git a/plugins/golang/test/test.go b/plugins/golang/test/test.go
new file mode 100644
index 00000000..c9544abc
--- /dev/null
+++ b/plugins/golang/test/test.go
@@ -0,0 +1,82 @@
+/* Test plugin.
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+package main
+
+/*
+#cgo pkg-config: nbdkit
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#define NBDKIT_API_VERSION 2
+#include <nbdkit-plugin.h>
+#include "wrappers.h"
+*/
+import "C"
+import "unsafe"
+import "libguestfs.org/nbdkit"
+
+//export pluginOpen
+func pluginOpen(readonly C.int) unsafe.Pointer {
+ nbdkit.Debug("golang code running in the .open callback")
+ // Cannot return a golang pointer, need to allocate a C obj.
+ return unsafe.Pointer(C.CString("hello"))
+}
+
+//export pluginClose
+func pluginClose(handle unsafe.Pointer) {
+ nbdkit.Debug("golang code running in the .close callback")
+ C.free(handle)
+}
+
+//export pluginGetSize
+func pluginGetSize(handle unsafe.Pointer) C.int64_t {
+ nbdkit.Debug("golang code running in the .get_size callback")
+ return 1024 * 1024
+}
+
+//export pluginPRead
+func pluginPRead(handle unsafe.Pointer, buf []byte, count C.uint32_t,
+ offset C.uint64_t, flags C.uint32_t) C.int {
+ nbdkit.Debug("golang code running in the .pread callback")
+ // XXX
+ return 0
+}
+
+func init_plugin() {
+ plugin.name = C.CString("test")
+ plugin.open = (*[0]byte)(C.open_wrapper)
+ plugin.close = (*[0]byte)(C.close_wrapper)
+ plugin.get_size = (*[0]byte)(C.get_size_wrapper)
+ plugin.pread = (*[0]byte)(C.pread_wrapper)
+}
diff --git a/plugins/golang/test/wrappers.go b/plugins/golang/test/wrappers.go
new file mode 100644
index 00000000..e7e0d3a9
--- /dev/null
+++ b/plugins/golang/test/wrappers.go
@@ -0,0 +1,85 @@
+/* Go wrappers
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+package main
+
+/*
+#cgo pkg-config: nbdkit
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#define NBDKIT_API_VERSION 2
+#include <nbdkit-plugin.h>
+#include "wrappers.h"
+
+extern void *pluginOpen ();
+void *
+open_wrapper (int readonly)
+{
+ return pluginOpen (readonly);
+}
+
+extern void pluginClose ();
+void
+close_wrapper (void *handle)
+{
+ pluginClose (handle);
+}
+
+extern int64_t pluginGetSize ();
+int64_t
+get_size_wrapper (void *handle)
+{
+ return pluginGetSize (handle);
+}
+
+extern int pluginPRead ();
+int
+pread_wrapper (void *handle, void *buf, uint32_t count, uint64_t offset,
+ uint32_t flags)
+{
+ return pluginPRead (handle, buf, count, offset, flags);
+}
+
+//extern int pluginPWrite ();
+//int
+//pwrite_wrapper (void *handle, const void *buf, uint32_t count, uint64_t offset,
+// uint32_t flags)
+//{
+// return pluginPWrite (handle, buf, count, offset, flags);
+//}
+
+// XXX ALL WRAPPERS
+
+*/
+import "C"
diff --git a/.gitignore b/.gitignore
index c44fb40d..7e71df35 100644
--- a/.gitignore
+++ b/.gitignore
@@ -71,6 +71,8 @@ plugins/*/*.3
/plugins/example4/nbdkit-example4-plugin
/plugins/eval/call.c
/plugins/eval/methods.c
+/plugins/golang/test/nbdkit-gotest-plugin.h
+/plugins/golang/test/nbdkit-gotest-plugin.so
/plugins/ocaml/nbdkit-ocamlexample-plugin.so
/plugins/rust/Cargo.lock
/plugins/rust/Cargo.toml
diff --git a/README b/README
index 05f1e060..1e05846d 100644
--- a/README
+++ b/README
@@ -148,6 +148,10 @@ For the Rust plugin:
- cargo (other dependencies will be downloaded at build time)
+To be able to write plugins in golang:
+
+ - go >= 1.5
+
For bash tab completion:
- bash-completion >= 1.99
--
2.25.0
11:27 a.m.
On Fri, Apr 10, 2020 at 02:51:52PM +0100, Richard W.M. Jones wrote:
Similar to C, OCaml and Rust, this is not a plugin per se. Instead
it's more of a method and set of tests around writing plugins in
golang. They are standalone programs that compile into shared objects
that nbdkit can then load (so there is no "go plugin" between nbdkit
and the user plugin, unlike in scripting languages like Perl).
Why did you choose this approach ?
Looking at the code below there's the general boilerplate that
will be approx the same for all plugins with cut+paste tedium
across projects, and then there's the "interesting" code in
test.go
The methods in test.go though look quite unappealing from a
Go programmer's POV, as the API contracts are full of CGo
types and unsafe pointers.
To make something that is attractive for Go programmers, I
think it needs to hide all the low level CGo stuff entirely.
Implementing a nbdkit plugin should require nothing more
that providing pure Go code that satisfies a well defined
Go "interface" type definition. There should not be any
copy+paste of example boilerplate, nor any use of CGo.
As an illustration, consider an interface and basic infrastructure:
package nbdkitplugin
type NBDKitFeature int
const (
NBDKitFeatureGetSize NBDKitFeature = iota
NBDKitFeaturePread
...other optional methods...
)
type NBDKitPlugin interface {
NBDGetFeatures() []NBDKitFeature
NBDOpen(readonly bool)
NBDClose()
NBDGetSize() int64
NBDPRead(count uint32, offset uint64, flags uint32) ([]byte)
....many other methods...
}
var plugin *C.struct_nbdkit_plugin
var pluginImpl NBDKitPlugin
func PluginInit(name string, impl NBDKitPlugin) {
pluginImpl = impl
features = impl.GetFeatures()
plugin.name = C.CString(name)
plugin.open = (*[0]byte)(C.open_wrapper)
plugin.close = (*[0]byte)(C.close_wrapper)
for _, feature := range features {
switch feature {
case NBDKitFeatureGetSize:
plugin.get_size = (*[0]byte)(C.get_size_wrapper)
case NBDKitFeaturePread:
plugin.pread = (*[0]byte)(C.pread_wrapper)
}
}
}
....all the methods like C.pread_wrapper/C.get_size_wrapper
need to invoke pluginImpl methods....
// This type implements all methods in NBDKitPlugin, with
// no-op impls. This means that people implementing plugins
// don't need to implement every method in the interface,
// only the few they care about
type PluginBase struct {
}
// We don't implement anytrhing by default
func (plugin *PluginBase) NBDGetFeatures() []NBDKitFeature {
return []NBDKitFeature{}
}
func (plugin *PluginBase) NBDOpen(readonly bool) {}
func (plugin *PluginBase) NBDClose() {}
func (plugin *PluginBase) NBDGetSize() int64 { return 0 }
}
func (plugin *PluginBase) NBDPRead(count uint32, offset uint64, flags uint32) []byte {
return []byte{}
}
... no-op impls of all other methods for NBDKitPlugin interface...
This code above would all be a standalone go module that is just
imported.
Now an impl of a plugin becomes just one single file
import (
libguestfs.org/nbdkitplugin
)
type TestPlugin struct {
nbdkitplugin.PluginBase // This provides no-op impls of all methods
...blah...
}
func NewTestPlugin() NBDKitPlugin {
return &TestPlugin{
....blah...
}
}
// This declares which methods we're actually implementing
func (plugin *TestPlugin) NBDGetFeatures() []NBDKitFeature {
return []NBDKitFeature{
NBDKitFeatureGetSize,
NBDKitFeaturePread,
}
}
func (plugin *TestPlugin) NBDOpen(readonly bool) {
nbdkit.Debug("golang code running in the .open callback")
}
func (plugin *TestPlugin) NBDClose() {
nbdkit.Debug("golang code running in the .close callback")
}
func (plugin *TestPlugin) NBDGetSize() int64 {
nbdkit.Debug("golang code running in the .get_size callback")
return 1024 * 1024
}
func (plugin *TestPlugin) NBDPRead(count uint32, offset uint64, flags uint32) []byte {
nbdkit.Debug("golang code running in the .pread callback")
return []byte{}
}
... don't need to implement any other methods, since PluginBase
satisfies the interface contract....
func init() {
nbdkitplugin.PluginInitialize("test", NewTestPlugin())
}
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
12:04 p.m.
On Wed, Apr 15, 2020 at 05:27:07PM +0100, Daniel P. Berrangé wrote:
On Fri, Apr 10, 2020 at 02:51:52PM +0100, Richard W.M. Jones wrote:
> Similar to C, OCaml and Rust, this is not a plugin per se. Instead
> it's more of a method and set of tests around writing plugins in
> golang. They are standalone programs that compile into shared objects
> that nbdkit can then load (so there is no "go plugin" between nbdkit
> and the user plugin, unlike in scripting languages like Perl).
Why did you choose this approach ?
Do you mean the design of constructing the .so file directly using
--buildmode=c-shared? I wasn't aware there was any other way to do
it. Should we use some kind of module loading instead?
Looking at the code below there's the general boilerplate that
will be approx the same for all plugins with cut+paste tedium
across projects, and then there's the "interesting" code in
test.go
The methods in test.go though look quite unappealing from a
Go programmer's POV, as the API contracts are full of CGo
types and unsafe pointers.
To make something that is attractive for Go programmers, I
think it needs to hide all the low level CGo stuff entirely.
Implementing a nbdkit plugin should require nothing more
that providing pure Go code that satisfies a well defined
Go "interface" type definition. There should not be any
copy+paste of example boilerplate, nor any use of CGo.
I don't disagree, I just have no idea how to achieve this.
As an illustration, consider an interface and basic infrastructure:
[...]
I'll have a go at changing it like this, but I don't see how this
changes any of the fundamental problems, but maybe I will understand
more once I've done that.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
12:43 p.m.
On Wed, Apr 15, 2020 at 06:04:26PM +0100, Richard W.M. Jones wrote:
On Wed, Apr 15, 2020 at 05:27:07PM +0100, Daniel P. Berrangé wrote:
> On Fri, Apr 10, 2020 at 02:51:52PM +0100, Richard W.M. Jones wrote:
> > Similar to C, OCaml and Rust, this is not a plugin per se. Instead
> > it's more of a method and set of tests around writing plugins in
> > golang. They are standalone programs that compile into shared objects
> > that nbdkit can then load (so there is no "go plugin" between nbdkit
> > and the user plugin, unlike in scripting languages like Perl).
>
> Why did you choose this approach ?
Do you mean the design of constructing the .so file directly using
--buildmode=c-shared? I wasn't aware there was any other way to do
it. Should we use some kind of module loading instead?
No, I was mostly meaning the "copylib" approach to plugins, where the
plugin author owns the whole stack.
I was just thinking of what the "ideal" would be from a Go programmers
POV, and that I think involves the least possible exposure to CGo and
the native nbdkit APIs.
> Looking at the code below there's the general boilerplate
that
> will be approx the same for all plugins with cut+paste tedium
> across projects, and then there's the "interesting" code in
> test.go
>
> The methods in test.go though look quite unappealing from a
> Go programmer's POV, as the API contracts are full of CGo
> types and unsafe pointers.
>
> To make something that is attractive for Go programmers, I
> think it needs to hide all the low level CGo stuff entirely.
>
> Implementing a nbdkit plugin should require nothing more
> that providing pure Go code that satisfies a well defined
> Go "interface" type definition. There should not be any
> copy+paste of example boilerplate, nor any use of CGo.
I don't disagree, I just have no idea how to achieve this.
> As an illustration, consider an interface and basic infrastructure:
[...]
I'll have a go at changing it like this, but I don't see how this
changes any of the fundamental problems, but maybe I will understand
more once I've done that.
I think the hairy CGo/libnbdkit interaction problems still exist in what
I'm describing. Mostly I'm just thinking about how they can be isolated
so that they owned by nbdkit maintainers in one place, and not copied
around, nor even seen by plugin authors.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1:02 p.m.
On Wed, Apr 15, 2020 at 06:43:52PM +0100, Daniel P. Berrangé wrote:
On Wed, Apr 15, 2020 at 06:04:26PM +0100, Richard W.M. Jones wrote:
> On Wed, Apr 15, 2020 at 05:27:07PM +0100, Daniel P. Berrangé wrote:
> > On Fri, Apr 10, 2020 at 02:51:52PM +0100, Richard W.M. Jones wrote:
> > > Similar to C, OCaml and Rust, this is not a plugin per se. Instead
> > > it's more of a method and set of tests around writing plugins in
> > > golang. They are standalone programs that compile into shared objects
> > > that nbdkit can then load (so there is no "go plugin" between
nbdkit
> > > and the user plugin, unlike in scripting languages like Perl).
> >
> > Why did you choose this approach ?
>
> Do you mean the design of constructing the .so file directly using
> --buildmode=c-shared? I wasn't aware there was any other way to do
> it. Should we use some kind of module loading instead?
No, I was mostly meaning the "copylib" approach to plugins, where the
plugin author owns the whole stack.
OK I see. In fact for the OCaml nbdkit bindings we provide a nice
OCaml library which the end user will link to their code to create the
final plugin. Ideally I want the same thing for Go too. (For Rust,
unfortunately, we didn't get this far yet, so basically Rust plugins
are programmed to the C API.)
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
8:57 a.m.
On Fri, Apr 10, 2020 at 02:51:51PM +0100, Richard W.M. Jones wrote:
Here is a patch that allows you to write nbdkit plugins in golang.
As
with C, OCaml and Rust, you can write a plugin in Go which compiles
directly to a .so file that can be loaded into golang, so in that
into *nbdkit*
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
1647
days inactive
1652
days old
6 comments
2 participants
participants (2)
-
Daniel P. Berrangé -
Richard W.M. Jones