9:52 a.m.
On Wed, Jan 04, 2023 at 06:14:34PM +0000, Richard W.M. Jones wrote:
This is an incomplete outline implementation for a libblkio plugin
for
nbdkit. At the moment it only supports reading the same ("capacity")
of the device, and not even reading or writing. I have some questions
about the libblkio API before I can complete the plugin (see below).
The idea here is that by connecting libblkio to NBD we can use the
existing set of scripting tools to script access to devices. For
example you could use Python to read or modify a device:
----------------------------------------------------------------------
$ nbdsh -c - <<'EOF'
from subprocess import *
# Run nbdkit-blkio-plugin.
h.connect_systemd_socket_activation ("nbdkit" "blkio",
"virtio-blk-vhost-user",
"path=unix.sock")
print("device size=%d", h.get_size())
# Dump the boot sector.
bootsect = h.pread(512, 0)
p = Popen("hexdump -C", shell=True, stdin=PIPE)
p.stdin.write(bootsect)
EOF
----------------------------------------------------------------------
So my questions and comments about libblkio:
Hi Rich,
This is cool! I've answered below:
(1) There is no way to know which properties are readable, writable,
and those which need to be set before or after blkio_connect (see
is_preconnect_property in the plugin). It should be possible to
introspect this information. Also might be nice to be able read a
list of all available properties.
libblkio doesn't support property introspection. The documentation
covers when each property can be read/written (e.g. read/write before
started and read-only after started).
Why is introspection needed?
(2) It would be nice if libblkio had a way to enable debugging and
call the caller back for debug messages. We could then redirect the
callbacks into the nbdkit debug API (nbdkit_debug()) where they would
eventually end up on stderr or syslog.
However don't send debug messages to stderr, or at least allow that
behaviour to be overridden.
libblkio has no debug messages. What stderr output did you get?
Errors are returned via blkio_get_error_msg().
(3) It seems like some drivers require pre-allocated memory regions,
and since some do that means we might as well implement this. It
also seems like some drivers require file-backed pre-allocated
memory regions, and so we might as well implement that too.
However what is not clear: does memfd_create(2) always allocate
sufficiently aligned memory regions, such that we never need to bother
reading the mem-region-alignment property?
I notice that the example:
https://gitlab.com/libblkio/libblkio/-/blob/main/examples/blkio-copy.c
just passes on this and calls blkio_alloc_mem_region(). Is that the
safest and easiest thing to do which will always work?
Yes, blkio_alloc_mem_region() is the safest and easiest way to allocate
correctly-aligned memory.
Regarding memfd_create(2), I don't believe the memory has any alignment
until it's mmapped and then it will be page-aligned. Normally page
alignment is sufficient for "buf-alignment" and
"mem-region-alignment",
but in theory a loop could be used to increase the size of the memfd to
achieve any required alignment. That's why blkio_alloc_mem_region() is
exists as an easy API that most people will use instead of allocating
memory regions on their own.
(4) As a first pass, I only want to bother implementing blocking mode.
It'll be slow, but it doesn't matter for this application. Also I've
chosen nbdkit's NBDKIT_THREAD_MODEL_SERIALIZE_ALL_REQUESTS so nbdkit
will serialise all requests (again, only for a very simple first pass).
Looking at: https://libblkio.gitlab.io/libblkio/blkio.html#performing-i-o
seems simple enough but:
(4a) What is the queue number? Always 0? Is it affected by num-entries?
Yes, the first queue is 0. "num-queues" sets the number of queues.
"num-entries" is a different property that some drivers have for sizing
rings.
(4b) It's unclear how completions work. If I set min=max=1,
will it
return after the whole operation has completed? Do I need to
call it again? What about if the request is very large, can it
get split?
Reading the example
https://gitlab.com/libblkio/libblkio/-/blob/main/examples/blkio-copy.c
it appears that requests cannot ever be split?
min=max=1 blocks until exactly 1 completion occurs. BTW I checked the
documentation to see if min_completions/max_completions are covered and
I'm not sure how to improve the documentation?
There is no need to call it again if you got the completion you were
waiting for.
The caller must honor the constraints given by the "max-segments",
"max-segment-len", and "max-transfer" properties. Requests are not
split.
Stefan