6:40 a.m.
On Jun 15 2022, "Richard W.M. Jones" <rjones(a)redhat.com> wrote:
On Wed, Jun 15, 2022 at 11:09:31AM +0100, Richard W.M. Jones wrote:
> This kernel setting limits the maximum request size on the queue.
>
> In my testing reading and writing files with the default [above] the
> kernel never got anywhere near sending multi-megabyte requests. In
> fact the largest request it sent was 128K, even when I did stuff like:
>
> # dd if=/dev/zero of=/tmp/mnt/zero bs=100M count=10
>
> 128K happens to be 2 x blk_queue_io_opt, but I need to do more testing
> to see if that relationship always holds.
The answer is apparently no. With minimum_io_size == 64K and
optimal_io_size == 256K, the server still only sees at most 128K
requests.
I have seen larger requests. Example (from the stats fiter):
WRITE request sizes:
131072 bytes: 66.1% of requests (8521)
9 bit aligned: 100.0% (8521)
10 bit aligned: 62.5% (5327)
11 bit aligned: 28.6% (2438)
12 bit aligned: 10.6% (907)
13 bit aligned: 6.0% (508)
14 bit aligned: 5.0% (426)
15+ bit-aligned: 4.9% (415)
262144 bytes: 9.2% of requests (1185)
9 bit aligned: 100.0% (1185)
10 bit aligned: 67.2% (796)
11 bit aligned: 48.8% (578)
12 bit aligned: 13.4% (159)
14 bit aligned: 11.9% (141)
15 bit aligned: 11.8% (140)
16 bit aligned: 11.7% (139)
17 bit aligned: 11.3% (134)
18 bit aligned: 5.8% (69)
19+ bit-aligned: 2.7% (32)
393216 bytes: 6.5% of requests (841)
9 bit aligned: 100.0% (841)
10 bit aligned: 64.2% (540)
11 bit aligned: 52.2% (439)
12 bit aligned: 17.5% (147)
13 bit aligned: 15.2% (128)
14 bit aligned: 14.6% (123)
15 bit aligned: 14.0% (118)
16 bit aligned: 13.8% (116)
17 bit aligned: 12.5% (105)
18 bit aligned: 6.4% (54)
19+ bit-aligned: 3.3% (28)
524288 bytes: 4.4% of requests (571)
9 bit aligned: 100.0% (571)
10 bit aligned: 55.2% (315)
11 bit aligned: 49.0% (280)
12 bit aligned: 34.7% (198)
13 bit aligned: 28.5% (163)
14 bit aligned: 26.8% (153)
16 bit aligned: 25.4% (145)
17 bit aligned: 22.4% (128)
18 bit aligned: 11.7% (67)
19 bit aligned: 6.3% (36)
20+ bit-aligned: 3.3% (19)
655360 bytes: 3.8% of requests (493)
9 bit aligned: 100.0% (493)
10 bit aligned: 57.0% (281)
11 bit aligned: 51.7% (255)
12 bit aligned: 46.0% (227)
13 bit aligned: 41.0% (202)
14 bit aligned: 40.4% (199)
16 bit aligned: 36.5% (180)
17 bit aligned: 32.5% (160)
18 bit aligned: 16.6% (82)
19 bit aligned: 9.3% (46)
20 bit aligned: 5.1% (25)
21+ bit-aligned: 1.8% (9)
786432 bytes: 2.1% of requests (270)
9 bit aligned: 100.0% (270)
10 bit aligned: 50.4% (136)
11 bit aligned: 47.4% (128)
12 bit aligned: 42.6% (115)
13 bit aligned: 34.1% (92)
14 bit aligned: 31.9% (86)
16 bit aligned: 31.1% (84)
17 bit aligned: 28.1% (76)
18 bit aligned: 14.8% (40)
19 bit aligned: 7.4% (20)
20+ bit-aligned: 3.3% (9)
1024 bytes: 1.8% of requests (238)
10 bit aligned: 100.0% (238)
11 bit aligned: 52.5% (125)
12 bit aligned: 25.2% (60)
13 bit aligned: 11.8% (28)
14 bit aligned: 5.0% (12)
17+ bit-aligned: 1.7% (4)
917504 bytes: 0.9% of requests (113)
9 bit aligned: 100.0% (113)
10 bit aligned: 61.9% (70)
11 bit aligned: 52.2% (59)
12 bit aligned: 50.4% (57)
14 bit aligned: 38.9% (44)
16 bit aligned: 38.1% (43)
17 bit aligned: 34.5% (39)
18 bit aligned: 18.6% (21)
19 bit aligned: 8.8% (10)
20+ bit-aligned: 3.5% (4)
1048576 bytes: 0.2% of requests (23)
9 bit aligned: 100.0% (23)
11 bit aligned: 56.5% (13)
16 bit aligned: 52.2% (12)
17 bit aligned: 43.5% (10)
19 bit aligned: 8.7% (2)
22 bit aligned: 4.3% (1)
other sizes: 4.9% of requests (631)
However, these are still far less than what I would have liked to see (I
was writing a number of huge files through ZFS, and had hoped to see
lots of 32 MB requests).
Although I still think we need to make these changes to nbd.ko, I
don't think this is going to solve the original problem of trying to
aggregate requests into the very large block sizes favoured by S3.
(nbdkit blocksize filter + a layer of caching seems like the way to go
for that)
Yeah.. that'd be quite a shame though. We'd be pretty much duplicating
the page cache, solely for the purpose of processing it in different
chunks than those produced by NBD.ko.
Best,
-Nikolaus
--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«