Hi,

I am observing some strange errors when using the Kernel's NBD driver with NBDkit.

On the kernel side, I see:

May 15 16:16:11 vostro.rath.org kernel: nbd0: detected capacity change from 0 to 104857600

May 15 16:16:11 vostro.rath.org kernel: nbd1: detected capacity change from 0 to 104857600

May 15 16:18:23 vostro.rath.org kernel: block nbd0: Possible stuck request 00000000ae5feee7: control (write@4836316160,32768B). Runtime 30 seconds

May 15 16:18:25 vostro.rath.org kernel: block nbd0: Possible stuck request 000000007094eddc: control (write@5372947456,10240B). Runtime 30 seconds

May 15 16:18:27 vostro.rath.org kernel: block nbd0: Suspicious reply 89 (status 0 flags 0)

May 15 16:18:31 vostro.rath.org kernel: block nbd0: Possible stuck request 0000000075f8b9bc: control (write@8057764864,32768B). Runtime 30 seconds

May 15 16:18:41 vostro.rath.org kernel: block nbd0: Possible stuck request 000000002d1b3e8b: control (write@14499979264,32768B). Runtime 30 seconds

[...]

And userspace ('zfs snapshot" in this instance) is stuck afterwards.

On the NBDkit side, there seemingly are write errors when replying back to the kernel:

$ nbdkit --unix /tmp/tmpi5o59_y_/nbd_socket_sb --foreground --filter=exitlast --filter=stats --threads 16 S3 size=50G bucket=nikratio-backup key=sb statsfile=/tmp/tmpi5o59_y_/stats_sb.txt object-size=32K &

$ nbd-client -unix /tmp/tmpi5o59_y_/nbd_socket_sb /dev/nbd2

Warning: the oldstyle protocol is no longer supported.

This method now uses the newstyle protocol with a default export

Negotiation: ..size = 51200MB

Connected /dev/nbd0

[....]

nbdkit: python.10: error: write reply: NBD_CMD_WRITE: Broken pipe

What's the best way to narrow down who's the culprit here (kernel vs NBD server)?

Best,

-Nikolaus