Re: [Libguestfs] ZFS-on-NBD
That looks like a missing piece that would have saved us a lot of headache.
At the time we were on linux 4.4 and 4.8. Thanks for sharing that.
Without deep diving through the impl, I wonder if that flag would also
protect against network buffer allocations in the kernel caused by network
reads/write in the userland process marked with that flag from entering the
IO flush paths as well. Because we are making network calls in our userland
block device driver to cloud storage, the socket buffers in the kernel can
also sometimes need to allocate additional buffer space and it was one
avenue we saw also trigger the IO flushing of dirty FS pages and re-enter
the FS kernel driver and block device drivers causing deadlock.
Kernel fs driver -> kernel block device driver -> userland block device
driver impl (makes network read/write calls) -> kernel network drivers
(decides to allocate additional socket buffer memory but triggers
reclaiming pages) -> kernel fs driver (wedges because locks already held)
-Shaun
On Fri, Sep 16, 2022 at 4:01 PM Nikolaus Rath <Nikolaus(a)rath.org> wrote:
On Sep 14 2022, Shaun McDowell <shaunjmcdowell(a)gmail.com>
wrote:
> It has been a few years since I've worked on it but there were a number
of
> gotchyas we had to overcome when running block devices in userspace and
> mounted on the same system. The largest of which is that the userland
> process needs to be extremely careful with memory allocations (ideally
all
> memory will be allocated at startup of the userland driver and memlocked)
> because you can quite easily cause deadlock on the system if your
userland
> block device driver needs to allocate memory at any point during an
> operation and the kernel decides it first needs to reclaim memory and
> chooses to clean out dirty filesystem buffer pages to satisfy the malloc.
> This causes the kernel to again try to re-enter the filesystem and block
> device kernel drivers to flush those dirty pages and that causes
deadlocks
> in the kernel as locks were already being held by the previous
> operation.
Is this not solved by PR_SET_IO_FLUSHER?
Best,
-Nikolaus
--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
_______________________________________________
Libguestfs mailing list
Libguestfs(a)redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Attachments:
- attachment.html (text/html — 3.1 KB)