Re: [Libguestfs] a question about multithreading with libguestfs
[Please keep replies on the mailing list so that others can benefit] On Tue, Oct 17, 2017 at 10:07:31PM +0300, Maxim Kozover wrote:
Hi Richard, Unfortunately I achieved a negative result, maybe you could help, please? . I'm using libguestfs-1.36.4 as a base since I changed for myself a bit some detection stuff that you recently moved from C to OCaml and I can't rewrite it immediately.
WIth vanilla 1.36.4 just changing fuse_loop to fuse_loop_mt gives almost immediate crash when the filesystem is accessed.
That's expected because plain 1.36 doesn't support multithreading, but ...
I've put the most recent gnulib, changed a bit bootstrap etc, put the patches from https://www.redhat.com/archives/libguestfs/2017-June/ msg00287.html and rebuilt.
... OK.
The system seems to work (with single-threaded fuse). Changed fuse_loop to fuse_loop_mt, rebuilt and fuse is stuck on the first call to appropriate fuse "system call" like mount_local_getattr in case of ls. It is stuck at guestfs_lstatns possibly at ACQUIRE_LOCK_FOR_CURRENT_SCOPE, while being called from fuse.
Can you get a stack trace from gdb. Use the command ‘t a a bt’ to show stacks from all threads at the same time.
What do you think could be a problem? Some lock already taken? Also, maybe less concern for me as I use fuse read-only, is the directory cache guarded?
Possibly not, I've not really looked at the fuse code w.r.t multithreading at all. Patches welcome as usual. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Hi Richard! Maybe the function guestfs_mount_local_run shouldn't ACQUIRE_LOCK_FOR_CURRENT_SCOPE as it doesn't talk with the daemon and sits in the loop? What do you think? If I remove it from guestfs_mount_local_run (in lib/action-1.c, don't know how to properly remove it from ml generator), fuse_loop_mt works, but I still don't understand how it worked with fuse_loop (single threaded) when guestfs_mount_local_run did ACQUIRE_LOCK_FOR_CURRENT_SCOPE. Unfortunately multiple ls -lR to the same directory tree lead to crash while it seems multiple ls -lR to different directory trees (if not repeated for a while) don't lead to crash, so I suspect directory cache in fuse without guard. Is there a quick way to disable the directory cache in fuse so I can see if it is directory cache that leads to crash or we have to look at other parts? Thanks much, Maxim. On Tue, Oct 17, 2017 at 11:25 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
[Please keep replies on the mailing list so that others can benefit]
On Tue, Oct 17, 2017 at 10:07:31PM +0300, Maxim Kozover wrote:
Hi Richard, Unfortunately I achieved a negative result, maybe you could help, please? . I'm using libguestfs-1.36.4 as a base since I changed for myself a bit some detection stuff that you recently moved from C to OCaml and I can't rewrite it immediately.
WIth vanilla 1.36.4 just changing fuse_loop to fuse_loop_mt gives almost immediate crash when the filesystem is accessed.
That's expected because plain 1.36 doesn't support multithreading, but ...
I've put the most recent gnulib, changed a bit bootstrap etc, put the patches from https://www.redhat.com/archives/libguestfs/2017-June/ msg00287.html and rebuilt.
... OK.
The system seems to work (with single-threaded fuse). Changed fuse_loop to fuse_loop_mt, rebuilt and fuse is stuck on the first call to appropriate fuse "system call" like mount_local_getattr in case of ls. It is stuck at guestfs_lstatns possibly at ACQUIRE_LOCK_FOR_CURRENT_ SCOPE, while being called from fuse.
Can you get a stack trace from gdb. Use the command ‘t a a bt’ to show stacks from all threads at the same time.
What do you think could be a problem? Some lock already taken? Also, maybe less concern for me as I use fuse read-only, is the directory cache guarded?
Possibly not, I've not really looked at the fuse code w.r.t multithreading at all. Patches welcome as usual.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~ rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Wed, Oct 18, 2017 at 01:33:49AM +0300, Maxim Kozover wrote:
Hi Richard! Maybe the function guestfs_mount_local_run shouldn't ACQUIRE_LOCK_FOR_CURRENT_SCOPE as it doesn't talk with the daemon and sits in the loop? What do you think?
The lock is needed for any access to the guestfs_h structure, so I'm pretty sure that is not safe.
If I remove it from guestfs_mount_local_run (in lib/action-1.c, don't know how to properly remove it from ml generator), fuse_loop_mt works, but I still don't understand how it worked with fuse_loop (single threaded) when guestfs_mount_local_run did ACQUIRE_LOCK_FOR_CURRENT_SCOPE. Unfortunately multiple ls -lR to the same directory tree lead to crash while it seems multiple ls -lR to different directory trees (if not repeated for a while) don't lead to crash, so I suspect directory cache in fuse without guard. Is there a quick way to disable the directory cache in fuse so I can see if it is directory cache that leads to crash or we have to look at other parts?
You could try removing the directory cache altogether from the FUSE code (it may run much more slowly). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Hi Richard! Finally it seems it works after removing ACQUIRE_LOCK_FOR_CURRENT_SCOPE in guestfs_mount_local_run and throwing away all directory cache code. Although I didn't try to mix fuse libguestfs client with other clients, I think guestfs_mount_local_run shouldn't take that specific lock as the function won't exit until umount and will sit in the loop. I think it does nothing libguestfs-related, only fuse loop activation. What I don't understand is how that lock in guestfs_mount_local_run didn't make problems with single threaded fuse_loop. I'll try to run it a bit more to see if any problems are seen. If everything is OK (hope not a false success as directory lists are slower and race condition maybe less likely), then it should be worth to see why directory cache made problems, maybe guard it with the same lock or different one? What do you think, Richard? Many thanks for your prompt help, Maxim. On Wed, Oct 18, 2017 at 1:39 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Oct 18, 2017 at 01:33:49AM +0300, Maxim Kozover wrote:
Hi Richard! Maybe the function guestfs_mount_local_run shouldn't ACQUIRE_LOCK_FOR_CURRENT_SCOPE as it doesn't talk with the daemon and sits in the loop? What do you think?
The lock is needed for any access to the guestfs_h structure, so I'm pretty sure that is not safe.
If I remove it from guestfs_mount_local_run (in lib/action-1.c, don't know how to properly remove it from ml generator), fuse_loop_mt works, but I still don't understand how it worked with fuse_loop (single threaded) when guestfs_mount_local_run did ACQUIRE_LOCK_FOR_CURRENT_SCOPE. Unfortunately multiple ls -lR to the same directory tree lead to crash while it seems multiple ls -lR to different directory trees (if not repeated for a while) don't lead to crash, so I suspect directory cache in fuse without guard. Is there a quick way to disable the directory cache in fuse so I can see if it is directory cache that leads to crash or we have to look at other parts?
You could try removing the directory cache altogether from the FUSE code (it may run much more slowly).
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~ rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
participants (2)
-
Maxim Kozover -
Richard W.M. Jones