Kernel driver I/O block size hinting
This is a follow-up to this thread: https://listman.redhat.com/archives/libguestfs/2022-June/thread.html#29210 about getting the kernel client (nbd.ko) to obey block size constraints sent by the NBD server: https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md#block-siz... I was sent this very interesting design document about the original intent behind the kernel's I/O limits: https://people.redhat.com/msnitzer/docs/io-limits.txt There are four or five kernel block layer settings we could usefully adjust, and there are three NBD block size constraints, and in my opinion there's not a very clear mapping between them. But I'll have a go at what I think we should do. - - - (1) Kernel physical_block_size & logical_block_size: The example given is of a hard disk with 4K physical sectors (AF) which can nevertheless emulate 512-byte sectors. In this case you'd set physical_block_size = 4K, logical_block_size = 512b. Data structures (partition tables, etc) should be aligned to physical_block_size to avoid unnecessary RMW cycles. But the fundamental until of I/O is logical_block_size. Current behaviour of nbd.ko is that logical_block_size == physical_block_size == the nbd-client "-b" option (default: 512 bytes, contradicting the documentation). I think we should set logical_block_size == physical_block_size == MAX (512, NBD minimum block size constraint). What should happen to the nbd-client -b option? (2) Kernel minimum_io_size: The documentation says this is the "preferred minimum unit for random I/O". Current behaviour of nbd.ko is this is not set. I think the NBD's preferred block size should map to minimum_io_size. (3) Kernel optimal_io_size: The documentation says this is the "[preferred] streaming I/O [size]". Current behaviour of nbd.ko is this is not set. NBD doesn't really have the concept of streaming vs random I/O, so we could either ignore this or set it to the same value as minimum_io_size. I have a kernel patch allowing nbd-client to set both minimum_io_size and optimal_io_size from userspace. (4) Kernel blk_queue_max_hw_sectors: This is documented as: "set max sectors for a request ... Enables a low level driver to set a hard upper limit, max_hw_sectors, on the size of requests." Current behaviour of nbd.ko is that we set this to 65536 (sectors? blocks?), which for 512b sectors is 32M. I think we could set this to MIN (32M, NBD maximum block size constraint), converting the result to sectors. - - - What do people think? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Jun 14 2022, "Richard W.M. Jones" <rjones@redhat.com> wrote:
This is a follow-up to this thread:
https://listman.redhat.com/archives/libguestfs/2022-June/thread.html#29210
about getting the kernel client (nbd.ko) to obey block size constraints sent by the NBD server:
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md#block-siz...
I was sent this very interesting design document about the original intent behind the kernel's I/O limits:
https://people.redhat.com/msnitzer/docs/io-limits.txt
There are four or five kernel block layer settings we could usefully adjust, and there are three NBD block size constraints, and in my opinion there's not a very clear mapping between them. But I'll have a go at what I think we should do.
- - -
(1) Kernel physical_block_size & logical_block_size: The example given is of a hard disk with 4K physical sectors (AF) which can nevertheless emulate 512-byte sectors. In this case you'd set physical_block_size = 4K, logical_block_size = 512b.
Data structures (partition tables, etc) should be aligned to physical_block_size to avoid unnecessary RMW cycles. But the fundamental until of I/O is logical_block_size.
Current behaviour of nbd.ko is that logical_block_size == physical_block_size == the nbd-client "-b" option (default: 512 bytes, contradicting the documentation).
I think we should set logical_block_size == physical_block_size == MAX (512, NBD minimum block size constraint).
Why the lower bound of 512?
What should happen to the nbd-client -b option?
Perhaps it should become the lower-bound (instead of the hardcoded 512)? That's assuming there is a reason for having a client-specified lower bound.
(2) Kernel minimum_io_size: The documentation says this is the "preferred minimum unit for random I/O".
Current behaviour of nbd.ko is this is not set.
I think the NBD's preferred block size should map to minimum_io_size.
(3) Kernel optimal_io_size: The documentation says this is the "[preferred] streaming I/O [size]".
Current behaviour of nbd.ko is this is not set.
NBD doesn't really have the concept of streaming vs random I/O, so we could either ignore this or set it to the same value as minimum_io_size.
I have a kernel patch allowing nbd-client to set both minimum_io_size and optimal_io_size from userspace.
(4) Kernel blk_queue_max_hw_sectors: This is documented as: "set max sectors for a request ... Enables a low level driver to set a hard upper limit, max_hw_sectors, on the size of requests."
Current behaviour of nbd.ko is that we set this to 65536 (sectors? blocks?), which for 512b sectors is 32M.
FWIW, on my 5.16 kernel, the default is 65 kB (according to /sys/block/nbdX/queue/max_sectors_kb x 512b).
I think we could set this to MIN (32M, NBD maximum block size constraint), converting the result to sectors.
I don't think that's right. Rather, it should be NBD's preferred block size. Setting this to the preferred block size means that NBD requests will be this large whenever there are enough sequential dirty pages, and that no requests will ever be larger than this. I think this is exactly what the NBD server would like to have. Settings this to the maximum block size would mean that NBD requests will exceed the preferred size whenever there are enough sequential dirty pages (while still obeying the maximum). This seems strictly worse. Unrelated to the proposed changes (all of which I think are technically correct), I am wondering if this will have much practical benefits. As far as I can tell, the kernel currently aligns NBD requests to the logical/physical block size rather than the size of the NBD request. Are there NBD servers that would benefit from the kernel honoring the preferred blocksize if the data is not also aligned to this blocksize? Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
On Tue, Jun 14, 2022 at 08:30:15PM +0100, Nikolaus Rath wrote:
On Jun 14 2022, "Richard W.M. Jones" <rjones@redhat.com> wrote:
I think we should set logical_block_size == physical_block_size == MAX (512, NBD minimum block size constraint).
Why the lower bound of 512?
I suspect the kernel can't handle sector sizes smaller than 512 bytes. By default the NBD protocol advises advertising a minimum size of 1 byte, and I'm almost certain setting logical_block_size == 1 would break everything.
What should happen to the nbd-client -b option?
Perhaps it should become the lower-bound (instead of the hardcoded 512)? That's assuming there is a reason for having a client-specified lower bound.
Right, I don't think there's a reason to continue with the -b option. I only use it to set -b 512 to work around the annoying default in older versions (which was 1024).
(4) Kernel blk_queue_max_hw_sectors: This is documented as: "set max sectors for a request ... Enables a low level driver to set a hard upper limit, max_hw_sectors, on the size of requests."
Current behaviour of nbd.ko is that we set this to 65536 (sectors? blocks?), which for 512b sectors is 32M.
FWIW, on my 5.16 kernel, the default is 65 kB (according to /sys/block/nbdX/queue/max_sectors_kb x 512b).
I have: $ cat /sys/devices/virtual/block/nbd0/queue/max_hw_sectors_kb 32768 (ie. 32 MB) which I think comes from the nbd module setting: blk_queue_max_hw_sectors(disk->queue, 65536); multiplied by 512b sectors.
I think we could set this to MIN (32M, NBD maximum block size constraint), converting the result to sectors.
I don't think that's right. Rather, it should be NBD's preferred block size.
Setting this to the preferred block size means that NBD requests will be this large whenever there are enough sequential dirty pages, and that no requests will ever be larger than this. I think this is exactly what the NBD server would like to have.
This kernel setting limits the maximum request size on the queue. In my testing reading and writing files with the default [above] the kernel never got anywhere near sending multi-megabyte requests. In fact the largest request it sent was 128K, even when I did stuff like: # dd if=/dev/zero of=/tmp/mnt/zero bs=100M count=10 128K happens to be 2 x blk_queue_io_opt, but I need to do more testing to see if that relationship always holds.
Settings this to the maximum block size would mean that NBD requests will exceed the preferred size whenever there are enough sequential dirty pages (while still obeying the maximum). This seems strictly worse.
Unrelated to the proposed changes (all of which I think are technically correct), I am wondering if this will have much practical benefits. As far as I can tell, the kernel currently aligns NBD requests to the logical/physical block size rather than the size of the NBD request. Are there NBD servers that would benefit from the kernel honoring the preferred blocksize if the data is not also aligned to this blocksize?
I'm not sure I parsed this. Can you give an example? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit
On Wed, Jun 15, 2022 at 11:09:31AM +0100, Richard W.M. Jones wrote:
This kernel setting limits the maximum request size on the queue.
In my testing reading and writing files with the default [above] the kernel never got anywhere near sending multi-megabyte requests. In fact the largest request it sent was 128K, even when I did stuff like:
# dd if=/dev/zero of=/tmp/mnt/zero bs=100M count=10
128K happens to be 2 x blk_queue_io_opt, but I need to do more testing to see if that relationship always holds.
The answer is apparently no. With minimum_io_size == 64K and optimal_io_size == 256K, the server still only sees at most 128K requests. Although I still think we need to make these changes to nbd.ko, I don't think this is going to solve the original problem of trying to aggregate requests into the very large block sizes favoured by S3. (nbdkit blocksize filter + a layer of caching seems like the way to go for that) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Jun 15 2022, "Richard W.M. Jones" <rjones@redhat.com> wrote:
On Wed, Jun 15, 2022 at 11:09:31AM +0100, Richard W.M. Jones wrote:
This kernel setting limits the maximum request size on the queue.
In my testing reading and writing files with the default [above] the kernel never got anywhere near sending multi-megabyte requests. In fact the largest request it sent was 128K, even when I did stuff like:
# dd if=/dev/zero of=/tmp/mnt/zero bs=100M count=10
128K happens to be 2 x blk_queue_io_opt, but I need to do more testing to see if that relationship always holds.
The answer is apparently no. With minimum_io_size == 64K and optimal_io_size == 256K, the server still only sees at most 128K requests.
I have seen larger requests. Example (from the stats fiter): WRITE request sizes: 131072 bytes: 66.1% of requests (8521) 9 bit aligned: 100.0% (8521) 10 bit aligned: 62.5% (5327) 11 bit aligned: 28.6% (2438) 12 bit aligned: 10.6% (907) 13 bit aligned: 6.0% (508) 14 bit aligned: 5.0% (426) 15+ bit-aligned: 4.9% (415) 262144 bytes: 9.2% of requests (1185) 9 bit aligned: 100.0% (1185) 10 bit aligned: 67.2% (796) 11 bit aligned: 48.8% (578) 12 bit aligned: 13.4% (159) 14 bit aligned: 11.9% (141) 15 bit aligned: 11.8% (140) 16 bit aligned: 11.7% (139) 17 bit aligned: 11.3% (134) 18 bit aligned: 5.8% (69) 19+ bit-aligned: 2.7% (32) 393216 bytes: 6.5% of requests (841) 9 bit aligned: 100.0% (841) 10 bit aligned: 64.2% (540) 11 bit aligned: 52.2% (439) 12 bit aligned: 17.5% (147) 13 bit aligned: 15.2% (128) 14 bit aligned: 14.6% (123) 15 bit aligned: 14.0% (118) 16 bit aligned: 13.8% (116) 17 bit aligned: 12.5% (105) 18 bit aligned: 6.4% (54) 19+ bit-aligned: 3.3% (28) 524288 bytes: 4.4% of requests (571) 9 bit aligned: 100.0% (571) 10 bit aligned: 55.2% (315) 11 bit aligned: 49.0% (280) 12 bit aligned: 34.7% (198) 13 bit aligned: 28.5% (163) 14 bit aligned: 26.8% (153) 16 bit aligned: 25.4% (145) 17 bit aligned: 22.4% (128) 18 bit aligned: 11.7% (67) 19 bit aligned: 6.3% (36) 20+ bit-aligned: 3.3% (19) 655360 bytes: 3.8% of requests (493) 9 bit aligned: 100.0% (493) 10 bit aligned: 57.0% (281) 11 bit aligned: 51.7% (255) 12 bit aligned: 46.0% (227) 13 bit aligned: 41.0% (202) 14 bit aligned: 40.4% (199) 16 bit aligned: 36.5% (180) 17 bit aligned: 32.5% (160) 18 bit aligned: 16.6% (82) 19 bit aligned: 9.3% (46) 20 bit aligned: 5.1% (25) 21+ bit-aligned: 1.8% (9) 786432 bytes: 2.1% of requests (270) 9 bit aligned: 100.0% (270) 10 bit aligned: 50.4% (136) 11 bit aligned: 47.4% (128) 12 bit aligned: 42.6% (115) 13 bit aligned: 34.1% (92) 14 bit aligned: 31.9% (86) 16 bit aligned: 31.1% (84) 17 bit aligned: 28.1% (76) 18 bit aligned: 14.8% (40) 19 bit aligned: 7.4% (20) 20+ bit-aligned: 3.3% (9) 1024 bytes: 1.8% of requests (238) 10 bit aligned: 100.0% (238) 11 bit aligned: 52.5% (125) 12 bit aligned: 25.2% (60) 13 bit aligned: 11.8% (28) 14 bit aligned: 5.0% (12) 17+ bit-aligned: 1.7% (4) 917504 bytes: 0.9% of requests (113) 9 bit aligned: 100.0% (113) 10 bit aligned: 61.9% (70) 11 bit aligned: 52.2% (59) 12 bit aligned: 50.4% (57) 14 bit aligned: 38.9% (44) 16 bit aligned: 38.1% (43) 17 bit aligned: 34.5% (39) 18 bit aligned: 18.6% (21) 19 bit aligned: 8.8% (10) 20+ bit-aligned: 3.5% (4) 1048576 bytes: 0.2% of requests (23) 9 bit aligned: 100.0% (23) 11 bit aligned: 56.5% (13) 16 bit aligned: 52.2% (12) 17 bit aligned: 43.5% (10) 19 bit aligned: 8.7% (2) 22 bit aligned: 4.3% (1) other sizes: 4.9% of requests (631) However, these are still far less than what I would have liked to see (I was writing a number of huge files through ZFS, and had hoped to see lots of 32 MB requests).
Although I still think we need to make these changes to nbd.ko, I don't think this is going to solve the original problem of trying to aggregate requests into the very large block sizes favoured by S3. (nbdkit blocksize filter + a layer of caching seems like the way to go for that)
Yeah.. that'd be quite a shame though. We'd be pretty much duplicating the page cache, solely for the purpose of processing it in different chunks than those produced by NBD.ko. Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
On Jun 15 2022, "Richard W.M. Jones" <rjones@redhat.com> wrote:
I think we could set this to MIN (32M, NBD maximum block size constraint), converting the result to sectors.
I don't think that's right. Rather, it should be NBD's preferred block size.
Setting this to the preferred block size means that NBD requests will be this large whenever there are enough sequential dirty pages, and that no requests will ever be larger than this. I think this is exactly what the NBD server would like to have.
This kernel setting limits the maximum request size on the queue.
Right. But why not limit it to the *preferred* blocksize of the NBD server? The kernel obviously does not care, and the NBD server obviously prefers this blocksize over the maximum block size.
In my testing reading and writing files with the default [above] the kernel never got anywhere near sending multi-megabyte requests.
Well, yes, but that shouldn't affect which value we should use, I think.
Unrelated to the proposed changes (all of which I think are technically correct), I am wondering if this will have much practical benefits. As far as I can tell, the kernel currently aligns NBD requests to the logical/physical block size rather than the size of the NBD request. Are there NBD servers that would benefit from the kernel honoring the preferred blocksize if the data is not also aligned to this blocksize?
I'm not sure I parsed this. Can you give an example?
No - I am asking for examples :-). My question is: in which scenario is it helpful for the NBD server to receive non-aligned requests of its preferred blocksize? Isn't that just as bad as receiving requests with a non-preferred blocksize? Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
participants (2)
-
Nikolaus Rath -
Richard W.M. Jones