Re: [Libguestfs] Communication issues between NBD driver and NBDKit server

On Sun, May 15, 2022 at 04:45:11PM +0100, Nikolaus Rath wrote: > Hi, > > I am observing some strange errors when using the Kernel's NBD driver with > NBDkit. > > On the kernel side, I see: > > May 15 16:16:11 vostro.rath.org kernel: nbd0: detected capacity change from 0 > to 104857600 > May 15 16:16:11 vostro.rath.org kernel: nbd1: detected capacity change from 0 > to 104857600 > May 15 16:18:23 vostro.rath.org kernel: block nbd0: Possible stuck request > 00000000ae5feee7: control (write@4836316160,32768B). Runtime 30 seconds > May 15 16:18:25 vostro.rath.org kernel: block nbd0: Possible stuck request > 000000007094eddc: control (write@5372947456,10240B). Runtime 30 seconds > May 15 16:18:27 vostro.rath.org kernel: block nbd0: Suspicious reply 89 (status > 0 flags 0) > May 15 16:18:31 vostro.rath.org kernel: block nbd0: Possible stuck request > 0000000075f8b9bc: control (write@8057764864,32768B). Runtime 30 seconds > May 15 16:18:41 vostro.rath.org kernel: block nbd0: Possible stuck request > 000000002d1b3e8b: control (write@14499979264,32768B). Runtime 30 seconds > [...] Does it really take over 30 seconds for nbdkit to respond? You might want to insert some debugging into the S3 plugin to see what stage of the request cycle is taking so long, although I'm going to guess it's the remote Amazon server itself.

It seems like you can adjust this timeout using the nbd-client -t flag (it calls ioctl(NBD_SET_TIMEOUT) in the kernel). If I understand the logic correctly, the nbd timeout is currently set to 0, which causes the default socket timeout to be used. Using the -t flag overrides this. So I guess try setting it larger and see if the problem goes away.