Re: [Libguestfs] Poor write performance with golang binding
On Wed, Feb 19, 2020 at 7:11 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
On Wed, Feb 19, 2020 at 03:00:11PM +0100, Csaba Henk wrote:
> [...]
> Then I do
>
> # pv /dev/sda2 | guest-xfer -d /dev/sda write mydisk.img
>
> I find that the ruby implementation produces a 24 MiB/s throughput, while
> the go one only 2 MiB/s.
>
> Note that the 'cat' operation (that writes device content to stdout)
> is reasonably fast with both language implementaitions (doing around
> 70 MiB/s).
>
> Why is this, how the Go binding could be improved?
TBH I've no idea. The bindings are meant to be very thin wrappers
around the C API, so I can't imagine that they should cause
performance penalties as large as you have observed.
Can you enable tracing in both (g.set_trace (true)) and make sure that
the operations being done are the same for both languages?
Rich.
Ok thanks for the idea. It helped to find out what's going on. And
(spoiler) there is nothing wrong here with the Go bindings.
I take data from stdout, through pipe. One read(2) from the pipe
fetches 64k data. `File.read` in Go directly exposes the syscall and
gives back 64k even if the argument buffer is larger (wrapping it in
bufio does not make difference). `File#read` in Ruby will read from
the file until the argument
buffer is filled.
Therefore (without custom buffering code, mapping stdin reads 1:1
to guestfs pwrites) `g.Pwrite_device` invocations in Go are capped
at 64k, while `g.pwrite_device` in Ruby will enjoy the the buffer size I
specify. And larger the buffer the better is the write performance (I
defaulted to 1m in my code, but giving bigger values, up to guestfs
proto limits is further improvement).
So TL;DR it's due to language idiosyncrasies, not because of
libguestfs.
Csaba