How to speed up Kernel Client - S3 plugin use-case
Hello, I am trying to improve performance of the scenario where the kernel's NBD client talks to NBDKit's S3 plugin. For me, the main bottleneck is currently due to the fact that the kernel aligns requests to only 512 B, no matter the blocksize reported by nbdkit. Using a 512 B object size is not feasible (due to latency and request overhead). However, with a larger object size there are two conflicting objectives: 1. To maximize parallelism (which is important to reduce the effects of connection latency), it's best to limit the size of the kernel's NBD requests to the object size. 2. To minimize un-aligned writes, it's best to allow arbitrarily large NBD requests, because the larger the requests the larger the amount of full blocks that are written. Unfortunately this means that all objects touched by the request are written sequentially. I see a number of ways to address that: 1. Change the kernel's NBD code to honor the blocksize reported by the NBD server. This would be ideal, but I don't feel up to making this happen. Theoretical solution only. 2. Change the S3 plugin to use multiple threads, so that it can upload multiple objects in parallel even when they're part of the same NBD request. The disadvantage is that this adds a second "layer" of threads, in addition to those started by NBDkit itself. 3. Change NBDkit itself to split up requests *and* distribute them to multiple threads. I believe this means changes to the core code because the blocksize filter can't dispatch requests to multiple threads. What do people think is the best way to proceed? Is there a fourth option that I might be missing? Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
On Mon, Jun 13, 2022 at 10:33:58AM +0100, Nikolaus Rath wrote:
Hello,
I am trying to improve performance of the scenario where the kernel's NBD client talks to NBDKit's S3 plugin.
For me, the main bottleneck is currently due to the fact that the kernel aligns requests to only 512 B, no matter the blocksize reported by nbdkit.
Using a 512 B object size is not feasible (due to latency and request overhead). However, with a larger object size there are two conflicting objectives:
1. To maximize parallelism (which is important to reduce the effects of connection latency), it's best to limit the size of the kernel's NBD requests to the object size.
2. To minimize un-aligned writes, it's best to allow arbitrarily large NBD requests, because the larger the requests the larger the amount of full blocks that are written. Unfortunately this means that all objects touched by the request are written sequentially.
I see a number of ways to address that:
1. Change the kernel's NBD code to honor the blocksize reported by the NBD server. This would be ideal, but I don't feel up to making this happen. Theoretical solution only.
This would be the ideal solution. I wonder how technically complicated it would be actually? AIUI you'd have to modify nbd-client to query the block limits from the server, which is the hardest part of this, but it's all userspace code. Then you'd pass those down to the kernel via the ioctl (see drivers/block/nbd.c:__nbd_ioctl). Then inside the kernel you'd call blk_queue_io_min & blk_queue_io_opt with the values (I'm not sure how you set the max request size, or if that's possible). See block/blk-settings.c for details of these functions. As a quick test you could try calling blk_queue_io_* in the kernel driver with hard-coded values, to see if that modifies the requests that are seen by nbdkit. Should give you some confidence before making the full change. BTW I notice that the kernel NBD driver always reports that it's a non-rotational device, ignoring the server setting ...
2. Change the S3 plugin to use multiple threads, so that it can upload multiple objects in parallel even when they're part of the same NBD request. The disadvantage is that this adds a second "layer" of threads, in addition to those started by NBDkit itself.
There are existing plugins which do this (see VDDK plugin).
3. Change NBDkit itself to split up requests *and* distribute them to multiple threads. I believe this means changes to the core code because the blocksize filter can't dispatch requests to multiple threads.
This would be a major change to nbdkit that would likely have unexpected side-effects everywhere.
What do people think is the best way to proceed? Is there a fourth option that I might be missing?
Best, -Nikolaus
Personally I think option (1) is the best here. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Jun 13 2022, "Richard W.M. Jones" <rjones@redhat.com> wrote:
On Mon, Jun 13, 2022 at 10:33:58AM +0100, Nikolaus Rath wrote:
Hello,
I am trying to improve performance of the scenario where the kernel's NBD client talks to NBDKit's S3 plugin.
For me, the main bottleneck is currently due to the fact that the kernel aligns requests to only 512 B, no matter the blocksize reported by nbdkit.
Using a 512 B object size is not feasible (due to latency and request overhead). However, with a larger object size there are two conflicting objectives:
1. To maximize parallelism (which is important to reduce the effects of connection latency), it's best to limit the size of the kernel's NBD requests to the object size.
2. To minimize un-aligned writes, it's best to allow arbitrarily large NBD requests, because the larger the requests the larger the amount of full blocks that are written. Unfortunately this means that all objects touched by the request are written sequentially.
I see a number of ways to address that:
1. Change the kernel's NBD code to honor the blocksize reported by the NBD server. This would be ideal, but I don't feel up to making this happen. Theoretical solution only.
This would be the ideal solution. I wonder how technically complicated it would be actually?
AIUI you'd have to modify nbd-client to query the block limits from the server, which is the hardest part of this, but it's all userspace code. Then you'd pass those down to the kernel via the ioctl (see drivers/block/nbd.c:__nbd_ioctl). Then inside the kernel you'd call blk_queue_io_min & blk_queue_io_opt with the values (I'm not sure how you set the max request size, or if that's possible). See block/blk-settings.c for details of these functions.
If it's only about getting the blocksize from the NBD server, then I certainly feel up to the task. However, nbd-client already has: -block-size block size -b Use a blocksize of "block size". Default is 1024; allowed values are ei‐ ther 512, 1024, 2048 or 4096 So my worry is that more complicated in-kernel changes will be needed to make other values work. In particular, nbd_is_valid_blksize() (in nbd.c) checks that the block size is less or equal to PAGE_SIZE. (I'm interested in 32 kB and 512 kB block sizes) Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
On Mon, Jun 13, 2022 at 11:58:11AM +0100, Nikolaus Rath wrote:
On Jun 13 2022, "Richard W.M. Jones" <rjones@redhat.com> wrote:
On Mon, Jun 13, 2022 at 10:33:58AM +0100, Nikolaus Rath wrote:
Hello,
I am trying to improve performance of the scenario where the kernel's NBD client talks to NBDKit's S3 plugin.
For me, the main bottleneck is currently due to the fact that the kernel aligns requests to only 512 B, no matter the blocksize reported by nbdkit.
Using a 512 B object size is not feasible (due to latency and request overhead). However, with a larger object size there are two conflicting objectives:
1. To maximize parallelism (which is important to reduce the effects of connection latency), it's best to limit the size of the kernel's NBD requests to the object size.
2. To minimize un-aligned writes, it's best to allow arbitrarily large NBD requests, because the larger the requests the larger the amount of full blocks that are written. Unfortunately this means that all objects touched by the request are written sequentially.
I see a number of ways to address that:
1. Change the kernel's NBD code to honor the blocksize reported by the NBD server. This would be ideal, but I don't feel up to making this happen. Theoretical solution only.
This would be the ideal solution. I wonder how technically complicated it would be actually?
AIUI you'd have to modify nbd-client to query the block limits from the server, which is the hardest part of this, but it's all userspace code. Then you'd pass those down to the kernel via the ioctl (see drivers/block/nbd.c:__nbd_ioctl). Then inside the kernel you'd call blk_queue_io_min & blk_queue_io_opt with the values (I'm not sure how you set the max request size, or if that's possible). See block/blk-settings.c for details of these functions.
If it's only about getting the blocksize from the NBD server, then I certainly feel up to the task.
However, nbd-client already has:
-block-size block size
-b Use a blocksize of "block size". Default is 1024; allowed values are ei‐ ther 512, 1024, 2048 or 4096
So my worry is that more complicated in-kernel changes will be needed to make other values work. In particular, nbd_is_valid_blksize() (in nbd.c) checks that the block size is less or equal to PAGE_SIZE.
(I'm interested in 32 kB and 512 kB block sizes)
This setting controls ioctl(nbd, NBD_SET_BLKSIZE,...) which inside the kernel calls: blk_queue_logical_block_size(nbd->disk->queue, blksize); blk_queue_physical_block_size(nbd->disk->queue, blksize); These functions are documented in block/blk-settings.c, but basically control the size of LBAs. For most devices that would be 512. (ISTR we changed the default in NBD a while back too, since 1024 caused problems for creating and reading some filesystems.) You can't really increase this setting to 2M or whatever S3 needs, because firstly it has to be smaller than the page size as you pointed out above, but mainly it'll radically change how filesystems get created since they use the block size as a basic unit to size other disk structures. In fact I wouldn't be surprised if most filesystems just don't function at all if the block size is massive. Nevertheless, tracing the code which sets this is instructive to see how you would adjust the same kernel code to set the minimum and preferred I/O settings via blk_queue_io_min / blk_queue_io_opt. These settings are separate from the block size (although must be multiples of the block size.) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Jun 13 2022, "Richard W.M. Jones" <rjones@redhat.com> wrote:
On Mon, Jun 13, 2022 at 11:58:11AM +0100, Nikolaus Rath wrote:
On Jun 13 2022, "Richard W.M. Jones" <rjones@redhat.com> wrote:
On Mon, Jun 13, 2022 at 10:33:58AM +0100, Nikolaus Rath wrote:
Hello,
I am trying to improve performance of the scenario where the kernel's NBD client talks to NBDKit's S3 plugin.
For me, the main bottleneck is currently due to the fact that the kernel aligns requests to only 512 B, no matter the blocksize reported by nbdkit.
Using a 512 B object size is not feasible (due to latency and request overhead). However, with a larger object size there are two conflicting objectives:
1. To maximize parallelism (which is important to reduce the effects of connection latency), it's best to limit the size of the kernel's NBD requests to the object size.
2. To minimize un-aligned writes, it's best to allow arbitrarily large NBD requests, because the larger the requests the larger the amount of full blocks that are written. Unfortunately this means that all objects touched by the request are written sequentially.
I see a number of ways to address that:
1. Change the kernel's NBD code to honor the blocksize reported by the NBD server. This would be ideal, but I don't feel up to making this happen. Theoretical solution only.
This would be the ideal solution. I wonder how technically complicated it would be actually?
AIUI you'd have to modify nbd-client to query the block limits from the server, which is the hardest part of this, but it's all userspace code. Then you'd pass those down to the kernel via the ioctl (see drivers/block/nbd.c:__nbd_ioctl). Then inside the kernel you'd call blk_queue_io_min & blk_queue_io_opt with the values (I'm not sure how you set the max request size, or if that's possible). See block/blk-settings.c for details of these functions.
If it's only about getting the blocksize from the NBD server, then I certainly feel up to the task.
However, nbd-client already has:
-block-size block size
-b Use a blocksize of "block size". Default is 1024; allowed values are ei‐ ther 512, 1024, 2048 or 4096
So my worry is that more complicated in-kernel changes will be needed to make other values work. In particular, nbd_is_valid_blksize() (in nbd.c) checks that the block size is less or equal to PAGE_SIZE.
(I'm interested in 32 kB and 512 kB block sizes)
This setting controls ioctl(nbd, NBD_SET_BLKSIZE,...) which inside the kernel calls:
blk_queue_logical_block_size(nbd->disk->queue, blksize); blk_queue_physical_block_size(nbd->disk->queue, blksize);
These functions are documented in block/blk-settings.c, but basically control the size of LBAs. For most devices that would be 512. (ISTR we changed the default in NBD a while back too, since 1024 caused problems for creating and reading some filesystems.)
You can't really increase this setting to 2M or whatever S3 needs, because firstly it has to be smaller than the page size as you pointed out above, but mainly it'll radically change how filesystems get created since they use the block size as a basic unit to size other disk structures. In fact I wouldn't be surprised if most filesystems just don't function at all if the block size is massive.
Nevertheless, tracing the code which sets this is instructive to see how you would adjust the same kernel code to set the minimum and preferred I/O settings via blk_queue_io_min / blk_queue_io_opt. These settings are separate from the block size (although must be multiples of the block size.)
Ah, this is helpful. Thank you for clarifying! I'll probably start with some experiments where I just hardcode a larger value in the kernel and see what happens. Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
On Mon, Jun 13, 2022 at 6:24 AM Richard W.M. Jones <rjones@redhat.com> wrote:
On Mon, Jun 13, 2022 at 10:33:58AM +0100, Nikolaus Rath wrote:
Hello,
I am trying to improve performance of the scenario where the kernel's NBD client talks to NBDKit's S3 plugin.
For me, the main bottleneck is currently due to the fact that the kernel aligns requests to only 512 B, no matter the blocksize reported by nbdkit.
Using a 512 B object size is not feasible (due to latency and request overhead). However, with a larger object size there are two conflicting objectives:
1. To maximize parallelism (which is important to reduce the effects of connection latency), it's best to limit the size of the kernel's NBD requests to the object size.
2. To minimize un-aligned writes, it's best to allow arbitrarily large NBD requests, because the larger the requests the larger the amount of full blocks that are written. Unfortunately this means that all objects touched by the request are written sequentially.
I see a number of ways to address that:
1. Change the kernel's NBD code to honor the blocksize reported by the NBD server. This would be ideal, but I don't feel up to making this happen. Theoretical solution only.
This would be the ideal solution. I wonder how technically complicated it would be actually?
AIUI you'd have to modify nbd-client to query the block limits from the server, which is the hardest part of this, but it's all userspace code. Then you'd pass those down to the kernel via the ioctl (see drivers/block/nbd.c:__nbd_ioctl). Then inside the kernel you'd call blk_queue_io_min & blk_queue_io_opt with the values (I'm not sure how you set the max request size, or if that's possible). See block/blk-settings.c for details of these functions.
Exactly this. The kernel just does what the client tells it to do, and the kernel can be configured for whatever blocksize. Unfortunately there's not a way for the server to advertise to the client what to do, you have to configure it on the client. Adding some code to userspace negotiation that happens is the right thing to do here to pull the blocksize, and then simply pass this into the configuration stuff in the nbd-client and it uses the appropriate netlink tag to set the blocksize.
As a quick test you could try calling blk_queue_io_* in the kernel driver with hard-coded values, to see if that modifies the requests that are seen by nbdkit. Should give you some confidence before making the full change.
BTW I notice that the kernel NBD driver always reports that it's a non-rotational device, ignoring the server setting ...
That I can fix easily, I'll get that done. Thanks, Josef
On Mon, Jun 13, 2022 at 01:25:39PM -0400, Josef Bacik wrote:
On Mon, Jun 13, 2022 at 6:24 AM Richard W.M. Jones <rjones@redhat.com> wrote:
On Mon, Jun 13, 2022 at 10:33:58AM +0100, Nikolaus Rath wrote:
Hello,
I am trying to improve performance of the scenario where the kernel's NBD client talks to NBDKit's S3 plugin.
For me, the main bottleneck is currently due to the fact that the kernel aligns requests to only 512 B, no matter the blocksize reported by nbdkit.
Using a 512 B object size is not feasible (due to latency and request overhead). However, with a larger object size there are two conflicting objectives:
1. To maximize parallelism (which is important to reduce the effects of connection latency), it's best to limit the size of the kernel's NBD requests to the object size.
2. To minimize un-aligned writes, it's best to allow arbitrarily large NBD requests, because the larger the requests the larger the amount of full blocks that are written. Unfortunately this means that all objects touched by the request are written sequentially.
I see a number of ways to address that:
1. Change the kernel's NBD code to honor the blocksize reported by the NBD server. This would be ideal, but I don't feel up to making this happen. Theoretical solution only.
This would be the ideal solution. I wonder how technically complicated it would be actually?
AIUI you'd have to modify nbd-client to query the block limits from the server, which is the hardest part of this, but it's all userspace code. Then you'd pass those down to the kernel via the ioctl (see drivers/block/nbd.c:__nbd_ioctl). Then inside the kernel you'd call blk_queue_io_min & blk_queue_io_opt with the values (I'm not sure how you set the max request size, or if that's possible). See block/blk-settings.c for details of these functions.
Exactly this. The kernel just does what the client tells it to do, and the kernel can be configured for whatever blocksize. Unfortunately there's not a way for the server to advertise to the client what to do, you have to configure it on the client. Adding some code to userspace negotiation that happens is the right thing to do here to pull the blocksize, and then simply pass this into the configuration stuff in the nbd-client and it uses the appropriate netlink tag to set the blocksize.
For context, the NBD protocol can now advertise during the initial handshake, minimum, preferred and maximum block sizes: https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md#block-siz... nbdkit (since 1.30) supports this, for example: $ nbdkit eval get_size='echo 256M' block_size='echo 64k 1M 32M' $ nbdinfo nbd://localhost protocol: newstyle-fixed without TLS export="": export-size: 268435456 (256M) uri: nbd://localhost:10809/ contexts: base:allocation is_rotational: false is_read_only: true can_cache: false can_df: true can_fast_zero: false can_flush: false can_fua: false can_multi_conn: false can_trim: false can_zero: false block_size_minimum: 65536 <--- block_size_preferred: 1048576 <--- block_size_maximum: 33554432 <--- Rich.
As a quick test you could try calling blk_queue_io_* in the kernel driver with hard-coded values, to see if that modifies the requests that are seen by nbdkit. Should give you some confidence before making the full change.
BTW I notice that the kernel NBD driver always reports that it's a non-rotational device, ignoring the server setting ...
That I can fix easily, I'll get that done. Thanks,
Josef
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Attached are the patches I'm experimenting with. I'm not sure if I'm missing something important. The first patch is for the kernel. This adds two netlink attributes NBD_ATTR_BLOCK_SIZE_{MIN,OPT} to set the minimum and preferred block size respectively. These appear to work, the values are reflected in /sys/devices/virtual/block/nbd0/queue/minimum_io_size, /sys/devices/virtual/block/nbd0/queue/optimal_io_size and in blkid: $ cat /sys/devices/virtual/block/nbd0/queue/minimum_io_size 16384 $ cat /sys/devices/virtual/block/nbd0/queue/optimal_io_size 65536 $ sudo blkid -i /dev/nbd0 /dev/nbd0: MINIMUM_IO_SIZE="16384" OPTIMAL_IO_SIZE="65536" PHYSICAL_SECTOR_SIZE="512" LOGICAL_SECTOR_SIZE="512" The second patch is for nbd and just hard-codes some values for testing purposes. The problem is that programs I've tried, including the kernel, don't respect these values very much. Certainly mkfs and the kernel doesn't seem to have a problem sending requests to the server which are smaller than the minimum we requested. It's hard to tell if it really has much effect or not. BTW I only modified the netlink interface, not the ioctl interface. Is it true that we've deprecated the ioctl interface to nbd.ko and prefer the netlink interface? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
participants (3)
-
Josef Bacik -
Nikolaus Rath -
Richard W.M. Jones