Libnbd asynchronous API with epoll
Hello, As part of the Rust bindings for Libnbd, I try to integrate the asynchronous (aio_*) functions with Tokio <https://docs.rs/tokio/latest/tokio/>, the most used asynchronous runtime in Rust. However, in its eventloop, Tokio uses epoll(7) instead of poll(2) (which is used internally in Libnbd). The difference is that poll(2) uses level-triggered notifications as aposed to epoll(7) which uses edge-triggered notifications. In short, the difference is that if one would wait for a file discriptor to be readable, epoll would block until the file descriptor changes from not readable to readable. So if the file descriptor is already readable then epoll would block until it becomes unreadable and readable again. I am not sure how to integrate Libnbd into an edge-triggered system. The following questions arises: - After calling aio_get_direction(3), can I know that reading/writing would actually block? - After calling for example aio_notify_read(3), can I know that the next reading from the file descriptor would block? If the answers to both of these questions are no, the only solution I can think of is to call poll(2) with a timeout of 0 as a non blocking check if the file descriptor is ready. Only if the call to poll(2) fails, I would use the epoll-driven eventloop to wait for changes. What do you think about this? Do you have an easier solution to integrate Libnbd with an edge-triggered eventloop? Best regards, Tage
On Wed, Jul 5, 2023 at 3:38 PM Tage Johansson <tage.j.lists@posteo.net> wrote:
As part of the Rust bindings for Libnbd, I try to integrate the asynchronous (aio_*) functions with Tokio <https://docs.rs/tokio/latest/tokio/>, the most used asynchronous runtime in Rust. However, in its eventloop, Tokio uses epoll(7) instead of poll(2) (which is used internally in Libnbd). The difference is that poll(2) uses level-triggered notifications as aposed to epoll(7) which uses edge-triggered notifications.
According to epoll(7) section "Level-triggered and edge-triggered" says: By contrast, when used as a level-triggered interface (the default, when EPOLLET is not specified), epoll is simply a faster poll(2), and can be used wherever the latter is used since it shares the same seman‐ tics. So you should not have any issue using epoll instead of poll.
- After calling aio_get_direction(3), can I know that reading/writing would actually block?
No, this is just the events that libnbd wants to get. - After calling for example aio_notify_read(3), can I know that the next
reading from the file descriptor would block?
No, you have to call again aio_get_direction() and poll again until the event happens. Nir
On 7/6/2023 7:06 PM, Nir Soffer wrote:
- After calling for example aio_notify_read(3), can I know that the next reading from the file descriptor would block?
No, you have to call again aio_get_direction() and poll again until the event happens.
Well, what I mean is: After calling aio_notify_read, if aio_get_direction returns AIO_DIRECTION_READ or AIO_DIRECTION_BOTH, can I know that the reading on the file descriptor actually blocked? Or might there be cases when aio_notify_read returns and the next direction includes a read and there is still more data to read on the file descriptor? I guess this is the case, but I must know or the client may hang unexpectedly. Best regards, Tage
On Fri, Jul 7, 2023 at 11:59 AM Tage Johansson <tage.j.lists@posteo.net> wrote:
On 7/6/2023 7:06 PM, Nir Soffer wrote:
- After calling for example aio_notify_read(3), can I know that the next reading from the file descriptor would block?
No, you have to call again aio_get_direction() and poll again until the event happens.
Well, what I mean is:
After calling aio_notify_read, if aio_get_direction returns AIO_DIRECTION_READ or AIO_DIRECTION_BOTH, can I know that the reading on the file descriptor actually blocked?
Yes - it never blocks.
Or might there be cases when aio_notify_read returns and the next direction includes a read and there is still more data to read on the file descriptor?
Sure it is expected that the socket is readable but more data will be available later...
I guess this is the case, but I must know or the client may hang unexpectedly.
Libnbd uses non-blocking socket so it will never hang.
(Sorry for the late reply, was a bit involved in a qemu bug last week ...) On Fri, Jul 07, 2023 at 08:58:50AM +0000, Tage Johansson wrote:
On 7/6/2023 7:06 PM, Nir Soffer wrote:
- After calling for example aio_notify_read(3), can I know that the next reading from the file descriptor would block?
No, you have to call again aio_get_direction() and poll again until the event happens.
Well, what I mean is:
After calling aio_notify_read, if aio_get_direction returns AIO_DIRECTION_READ or AIO_DIRECTION_BOTH, can I know that the reading on the file descriptor actually blocked? Or might there be cases when aio_notify_read returns and the next direction includes a read and there is still more data to read on the file descriptor?
It's probably best to have a look at the generated code, but it works like this: - aio_notify_read is the program telling libnbd that the socket is readable. ie. The socket won't block, one byte at least is available to read. - aio_get_direction is libnbd telling the program that we are in a state where we are expecting to read or write (or either) to the socket. As an example, let's say we have just connected to the NBD server. We expect the NBD server to send a magic string, here: https://gitlab.com/nbdkit/libnbd/-/blob/20dceca48623835c3e8a195ef4135cce5e33... For example: $ nbdkit -s null NBDMAGICIHAVEOPT <- magic string sent by the server But in this case let's say the server is slow to send the magic string. The call to recv_into_rbuf -> h->sock->ops->recv -> recv returns EWOULDBLOCK (recv_into_rbuf returns 1). Then we stay in the same state (MAGIC.RECV_MAGIC), and return to the main program. At this point if the main program calls nbd_aio_get_direction it would return AIO_DIRECTION_READ. We are expecting to read the socket next. A write on the socket is not possible in this state (so we will not return AIO_DIRECTION_BOTH). I hope that's a bit clearer. I'm not sure about the epoll question. I haven't thought about it deeply, but it seems as if edge-triggered notification might be a problem, but you could keep track of this from the main program with an extra variable, turning edge-triggered into level-triggered. Rich.
I guess this is the case, but I must know or the client may hang unexpectedly.
Best regards,
Tage
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Sun, Jul 09, 2023 at 11:01:47AM +0100, Richard W.M. Jones wrote:
(Sorry for the late reply, was a bit involved in a qemu bug last week ...)
On Fri, Jul 07, 2023 at 08:58:50AM +0000, Tage Johansson wrote:
On 7/6/2023 7:06 PM, Nir Soffer wrote:
- After calling for example aio_notify_read(3), can I know that the next reading from the file descriptor would block?
No, you have to call again aio_get_direction() and poll again until the event happens.
Well, what I mean is:
After calling aio_notify_read, if aio_get_direction returns AIO_DIRECTION_READ or AIO_DIRECTION_BOTH, can I know that the reading on the file descriptor actually blocked? Or might there be cases when aio_notify_read returns and the next direction includes a read and there is still more data to read on the file descriptor?
It's probably best to have a look at the generated code, but it works like this:
- aio_notify_read is the program telling libnbd that the socket is readable. ie. The socket won't block, one byte at least is available to read.
- aio_get_direction is libnbd telling the program that we are in a state where we are expecting to read or write (or either) to the socket.
As an example, let's say we have just connected to the NBD server. We expect the NBD server to send a magic string, here:
https://gitlab.com/nbdkit/libnbd/-/blob/20dceca48623835c3e8a195ef4135cce5e33...
For example:
$ nbdkit -s null NBDMAGICIHAVEOPT <- magic string sent by the server
But in this case let's say the server is slow to send the magic string.
The call to recv_into_rbuf -> h->sock->ops->recv -> recv returns EWOULDBLOCK (recv_into_rbuf returns 1). Then we stay in the same state (MAGIC.RECV_MAGIC), and return to the main program.
At this point if the main program calls nbd_aio_get_direction it would return AIO_DIRECTION_READ. We are expecting to read the socket next. A write on the socket is not possible in this state (so we will not return AIO_DIRECTION_BOTH).
... and to continue the thought: If the program now detects that the socket is readable using select/poll/epoll/etc, it can call nbd_aio_notify_read. This re-enters the state machine, reading from the socket, getting some data (hopefully), and moving the state machine on until it blocks again, at which point nbd_aio_notify_read returns. Then you'd call nbd_aio_get_direction again to find out if we are blocked expecting a read, write (or either), and the cycle continues. Rich.
I hope that's a bit clearer.
I'm not sure about the epoll question. I haven't thought about it deeply, but it seems as if edge-triggered notification might be a problem, but you could keep track of this from the main program with an extra variable, turning edge-triggered into level-triggered.
Rich.
I guess this is the case, but I must know or the client may hang unexpectedly.
Best regards,
Tage
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
participants (3)
-
Nir Soffer -
Richard W.M. Jones -
Tage Johansson