5:08 a.m.
I was going through the code and realized io_uring is similar to libev in
the way that we listen to the io events ourselves but the difference is
when we encounter the event we read it ourselves(io_uring), whereas in the
case of libev we ask libnbd to read it for us. We notify that the
connection is readable using ai_notify_read and the command is ran by the
state machine the command was queued by an earlier call to nbd_aio_pread.
My question is if we read from the socket using (let's say read[1]) what do
we need to do to inform the state machine we have done so, so that it can
function properly as I am assuming nbd_aio_pread modifies the state in some
way on enqueue and completion
Thanks and regards,
Abhay
---
[1]: io_uring has read but not recv in kernel 5.5
6:08 a.m.
On Fri, Jun 18, 2021 at 03:38:16PM +0530, Abhay Raj Singh wrote:
I was going through the code and realized io_uring is similar to
libev in the way that we listen to the io events ourselves but the
difference is when we encounter the event we read it
ourselves(io_uring), whereas in the case of libev we ask libnbd to
read it for us. We notify that the connection is readable using
aio_notify_read and the command is ran by the state machine the
command was queued by an earlier call to nbd_aio_pread.
The current "nbd_aio_*" API isn't really designed around completion
events. You're thinking about the right thing: what would an nbd_*
API look like that could be implemented using io_uring?
I find it useful to write some example programs using a proposed new
API (obviously not programs that can be compiled) to explore what API
could work.
My question is if we read from the socket using (let's say
read[1])
what do we need to do to inform the state machine we have done so,
so that it can function properly as I am assuming nbd_aio_pread
modifies the state in some way on enqueue and completion
The answer will be in the code :-) I don't remember all the exact
details of how it works now. The generated files are probably the
best place to start reading.
[1]: io_uring has read but not recv in kernel 5.5
The main technical reason we currently use send/recv instead of
write/read is so we can pass the flags MSG_NOSIGNAL and MSG_MORE.
MSG_MORE is an optimization so we don't have to use it. MSG_NOSIGNAL
is necessary, but at a first pass we could ignore it until we work out
what to do with it.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
7:44 a.m.
Another way to proceed here could be to write a from-scratch version
of nbdcopy using io_uring, to get a feel for the "shape" of what that
would look like. Then translate back from that into an NBD API
proposal.
For simplicity in implementation:
- Only implement NBD oldstyle protocol, which is very simple. You
can find it documented here:
https://gitlab.com/nbdkit/nbdkit/-/blob/dd894ae009c16810605252ee6996d329c...
nbdkit supports oldstyle if you add the -o option.
- Only do NBD server -> NBD server copies, forget about local files,
pipes, etc.
- Don't implement any of the options, synchronous mode, etc. Just
hard-code all the settings.
I'm interested in seeing what this would look like. Also it might
answer a few questions I have:
- Do we need to use threads in nbdcopy if we have io_uring?
- How many io_urings should we use?
- Can we avoid kernel-userspace copies?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
8:34 a.m.
Thanks! I too feel this would be the best approach.
Another way to proceed here could be to write a from-scratch version
of nbdcopy using io_uring, to get a feel for the "shape" of what that
would look like. Then translate back from that into an NBD API
proposal.
- Can we avoid kernel-userspace copies?
I implemented cp like
tool using io_uring and read+write, there was
only one "copy_user_enhanced_fast_string" in io_uring based one's
(flame graph) so I assume this should work.
Regards,
Abhay
1217
days inactive
1217
days old
3 comments
2 participants
participants (2)
-
Abhay Raj Singh -
Richard W.M. Jones