Show how to read entire image using the simple synchronous API and how and the high performance asynchronous API. For simplicity, the example do not use extents, do not sparsify the image, and copy the image to only to stdout. This make it easy to evaluate the Go bindings performance. The aio_copy example includes an interesting ordering queue, ensuring that asynchronous reads are written in the right order even if they completed out of order. This allows using the fast asynchronous API even when the output does not support seek, and may perform better even if the output does support seek when using rotational disks. The aio_copy example does not use AioBuffer in the normal way to avoid unwanted copy when AioPread completes. Instead, we use a Go allocated buffer, in the same way we pass a Go allocated buffer to libnbd in the synchronous API. This usage is unsafe but required for getting decent performance. Testing show that we get better performance than nbdcopy with the defaults, or the same performance if we tune nbdcopy to use the one connection and 4 requests, and drop the data using null: output. All tests use 6 GiB fully allocated zero image created with: $ dd if=/dev/zero bs=1M count=6144 of=zero-6g.raw The image is served using qemu-nbd: $ qemu-nbd -r -t -e0 -k /tmp/nbd.sock --cache=none --aio=native -f raw zero-6g.raw $ hyperfine "simple_copy/simple_copy $URL >/dev/null" \ "nbdcopy --synchronous --allocated $URL /dev/null" \ "nbdcopy --request-size $((2048*1024)) --synchronous --allocated $URL /dev/null" Benchmark 1: simple_copy/simple_copy nbd+unix:///?socket=/tmp/nbd.sock >/dev/null Time (mean ± σ): 3.210 s ± 0.065 s [User: 0.275 s, System: 0.836 s] Range (min … max): 3.117 s … 3.298 s 10 runs Benchmark 2: nbdcopy --synchronous --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null Time (mean ± σ): 4.469 s ± 0.019 s [User: 0.295 s, System: 0.948 s] Range (min … max): 4.447 s … 4.510 s 10 runs Benchmark 3: nbdcopy --request-size 2097152 --synchronous --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null Time (mean ± σ): 3.266 s ± 0.012 s [User: 0.216 s, System: 0.732 s] Range (min … max): 3.244 s … 3.286 s 10 runs Summary 'simple_copy/simple_copy nbd+unix:///?socket=/tmp/nbd.sock >/dev/null' ran 1.02 ± 0.02 times faster than 'nbdcopy --request-size 2097152 --synchronous --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null' 1.39 ± 0.03 times faster than 'nbdcopy --synchronous --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null' $ hyperfine "aio_copy/aio_copy $URL >/dev/null" \ "nbdcopy --allocated $URL /dev/null" \ "nbdcopy --allocated $URL null:" \ "nbdcopy --connections 1 --requests 4 --allocated $URL null:" Benchmark 1: aio_copy/aio_copy nbd+unix:///?socket=/tmp/nbd.sock >/dev/null Time (mean ± σ): 2.013 s ± 0.035 s [User: 0.410 s, System: 0.877 s] Range (min … max): 1.966 s … 2.060 s 10 runs Benchmark 2: nbdcopy --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null Time (mean ± σ): 4.501 s ± 0.025 s [User: 0.287 s, System: 0.949 s] Range (min … max): 4.449 s … 4.532 s 10 runs Benchmark 3: nbdcopy --allocated nbd+unix:///?socket=/tmp/nbd.sock null: Time (mean ± σ): 2.422 s ± 0.018 s [User: 0.520 s, System: 1.772 s] Range (min … max): 2.404 s … 2.470 s 10 runs Benchmark 4: nbdcopy --connections 1 --requests 4 --allocated nbd+unix:///?socket=/tmp/nbd.sock null: Time (mean ± σ): 2.019 s ± 0.009 s [User: 0.270 s, System: 0.845 s] Range (min … max): 2.008 s … 2.033 s 10 runs Summary 'aio_copy/aio_copy nbd+unix:///?socket=/tmp/nbd.sock >/dev/null' ran 1.00 ± 0.02 times faster than 'nbdcopy --connections 1 --requests 4 --allocated nbd+unix:///?socket=/tmp/nbd.sock null:' 1.20 ± 0.02 times faster than 'nbdcopy --allocated nbd+unix:///?socket=/tmp/nbd.sock null:' 2.24 ± 0.04 times faster than 'nbdcopy --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null' Signed-off-by: Nir Soffer <nsoffer(a)redhat.com&gt; --- Chagnes in v2: - Fix error handling: panic if aio_pread completion callback *error is non-zero. - Create AioPreadOptargs struct instead of pointer. This may save unneeded allocation per read and is little bit simpler. V1 was here: https://listman.redhat.com/archives/libguestfs/2022-January/msg00165.html golang/examples/Makefile.am | 26 ++- golang/examples/aio_copy/aio_copy.go | 198 +++++++++++++++++++++ golang/examples/aio_copy/go.mod | 4 + golang/examples/simple_copy/go.mod | 4 + golang/examples/simple_copy/simple_copy.go | 93 ++++++++++ 5 files changed, 323 insertions(+), 2 deletions(-) create mode 100644 golang/examples/aio_copy/aio_copy.go create mode 100644 golang/examples/aio_copy/go.mod create mode 100644 golang/examples/simple_copy/go.mod create mode 100644 golang/examples/simple_copy/simple_copy.go diff --git a/golang/examples/Makefile.am b/golang/examples/Makefile.am index bfeee245..90f9fbf8 100644 --- a/golang/examples/Makefile.am +++ b/golang/examples/Makefile.am @@ -16,27 +16,49 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA include $(top_srcdir)/subdir-rules.mk EXTRA_DIST = \ LICENSE-FOR-EXAMPLES \ get_size/go.mod \ get_size/get_size.go \ read_first_sector/go.mod \ read_first_sector/read_first_sector.go \ + simple_copy/go.mod \ + simple_copy/simple_copy.go \ + aio_copy/go.mod \ + aio_copy/aio_copy.go \ $(NULL) if HAVE_GOLANG -noinst_SCRIPTS = get_size/get_size read_first_sector/read_first_sector +noinst_SCRIPTS = \ + get_size/get_size \ + read_first_sector/read_first_sector \ + simple_copy/simple_copy \ + aio_copy/aio_copy \ + $(NULL) get_size/get_size: get_size/get_size.go cd get_size && \ $(abs_top_builddir)/run go build -o get_size read_first_sector/read_first_sector: read_first_sector/read_first_sector.go cd read_first_sector && \ $(abs_top_builddir)/run go build -o read_first_sector +simple_copy/simple_copy: simple_copy/simple_copy.go + cd simple_copy && \ + $(abs_top_builddir)/run go build -o simple_copy + +aio_copy/aio_copy: aio_copy/aio_copy.go + cd aio_copy && \ + $(abs_top_builddir)/run go build -o aio_copy + endif HAVE_GOLANG -CLEANFILES += get_size/get_size read_first_sector/read_first_sector +CLEANFILES += \ + get_size/get_size \ + read_first_sector/read_first_sector \ + simple_copy/simple_copy \ + aio_copy/aio_copy \ + $(NULL) diff --git a/golang/examples/aio_copy/aio_copy.go b/golang/examples/aio_copy/aio_copy.go new file mode 100644 index 00000000..b6f5def1 --- /dev/null +++ b/golang/examples/aio_copy/aio_copy.go @@ -0,0 +1,198 @@ +/* libnbd example + * Copyright (C) 2013-2022 Red Hat Inc. + * Examples are under a permissive BSD-like license. See also + * golang/examples/LICENSE-For-EXAMPLES + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +// Copy image from NBD URI to stdout. +// +// Example: +// +// ./aio_copy nbd+unix:///?socket=/tmp.nbd >/dev/null +// +package main + +import ( + "container/list" + "flag" + "os" + "sync" + "syscall" + "unsafe" + + "libguestfs.org/libnbd" +) + +var ( + // These options give best performance with fast NVMe drive. + requestSize = flag.Uint("request-size", 256*1024, "maximum request size in bytes") + requests = flag.Uint("requests", 4, "maximum number of requests in flight") + + h *libnbd.Libnbd + + // Keeping commands in a queue ensures commands are written in the right + // order, even if they complete out of order. This allows parallel reads + // with non-seekable output. + queue list.List + + // Buffer pool allocating buffers as needed and reusing them. + bufPool = sync.Pool{ + New: func() interface{} { + return make([]byte, *requestSize) + }, + } +) + +// command keeps state of single AioPread call while the read is handled by +// libnbd, until the command reach the front of the queue and can be writen to +// the output. +type command struct { + buf []byte + length uint + ready bool +} + +func main() { + flag.Parse() + + var err error + + h, err = libnbd.Create() + if err != nil { + panic(err) + } + defer h.Close() + + err = h.ConnectUri(flag.Arg(0)) + if err != nil { + panic(err) + } + + size, err := h.GetSize() + if err != nil { + panic(err) + } + + var offset uint64 + + for offset < size || queue.Len() > 0 { + + for offset < size && inflightRequests() < *requests { + length := *requestSize + if size-offset < uint64(length) { + length = uint(size - offset) + } + startRead(offset, length) + offset += uint64(length) + } + + waitForCompletion() + + for readReady() { + finishRead() + } + } +} + +func inflightRequests() uint { + n, err := h.AioInFlight() + if err != nil { + panic(err) + } + return n +} + +func waitForCompletion() { + start := inflightRequests() + + for { + _, err := h.Poll(-1) + if err != nil { + panic(err) + } + + if inflightRequests() < start { + break // A read completed. + } + } +} + +func startRead(offset uint64, length uint) { + buf := bufPool.Get().([]byte) + + // Keep buffer in command so we can put it back into the pool when the + // command completes. + cmd := &command{buf: buf, length: length} + + // Create aio buffer from pool buffer to avoid unneeded allocation for + // every read, and unneeded copy when completing the read. + abuf := libnbd.AioBuffer{P: unsafe.Pointer(&buf[0]), Size: length} + + args := libnbd.AioPreadOptargs{ + CompletionCallbackSet: true, + CompletionCallback: func(error *int) int { + if *error != 0 { + // This is not documented, but *error is errno value translated + // from the the NBD server error. + err := syscall.Errno(*error).Error() + panic(err) + } + cmd.ready = true + return 1 + }, + } + + _, err := h.AioPread(abuf, offset, &args) + if err != nil { + panic(err) + } + + queue.PushBack(cmd) +} + +func readReady() bool { + return queue.Len() > 0 && queue.Front().Value.(*command).ready +} + +func finishRead() { + e := queue.Front() + queue.Remove(e) + + cmd := e.Value.(*command) + b := cmd.buf[:cmd.length] + + _, err := os.Stdout.Write(b) + if err != nil { + panic(err) + } + + bufPool.Put(cmd.buf) +} diff --git a/golang/examples/aio_copy/go.mod b/golang/examples/aio_copy/go.mod new file mode 100644 index 00000000..074fabf7 --- /dev/null +++ b/golang/examples/aio_copy/go.mod @@ -0,0 +1,4 @@ +module main + +replace libguestfs.org/libnbd => ../../ +require libguestfs.org/libnbd v1.11.5 diff --git a/golang/examples/simple_copy/go.mod b/golang/examples/simple_copy/go.mod new file mode 100644 index 00000000..074fabf7 --- /dev/null +++ b/golang/examples/simple_copy/go.mod @@ -0,0 +1,4 @@ +module main + +replace libguestfs.org/libnbd => ../../ +require libguestfs.org/libnbd v1.11.5 diff --git a/golang/examples/simple_copy/simple_copy.go b/golang/examples/simple_copy/simple_copy.go new file mode 100644 index 00000000..e8fa1f76 --- /dev/null +++ b/golang/examples/simple_copy/simple_copy.go @@ -0,0 +1,93 @@ +/* libnbd example + * Copyright (C) 2013-2022 Red Hat Inc. + * Examples are under a permissive BSD-like license. See also + * golang/examples/LICENSE-For-EXAMPLES + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * * Neither the name of Red Hat nor the names of its contributors may be + * used to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +// Copy image from NBD URI to stdout. +// +// Example: +// +// ./simple_copy nbd+unix:///?socket=/tmp.nbd >/dev/null +// +package main + +import ( + "flag" + "os" + + "libguestfs.org/libnbd" +) + +var ( + requestSize = flag.Uint("buffer-size", 2048*1024, "maximum request size in bytes") +) + +func main() { + flag.Parse() + + h, err := libnbd.Create() + if err != nil { + panic(err) + } + defer h.Close() + + err = h.ConnectUri(flag.Arg(0)) + if err != nil { + panic(err) + } + + size, err := h.GetSize() + if err != nil { + panic(err) + } + + buf := make([]byte, *requestSize) + var offset uint64 + + for offset < size { + if size-offset < uint64(len(buf)) { + buf = buf[:offset-size] + } + + err = h.Pread(buf, offset, nil) + if err != nil { + panic(err) + } + + _, err := os.Stdout.Write(buf) + if err != nil { + panic(err) + } + + offset += uint64(len(buf)) + } +} -- 2.34.1