Re: [Libguestfs] [PATCH nbdkit 2/2] allocators: sparse: Split the highly contended mutex

We've long known that nbdkit-memory-plugin with the default sparse array allocator suffers because a global lock must be taken whenever any read or write operation is performed. This commit aims to safely improve performance by converting the lock into a read-write lock.

+++ b/common/allocators/sparse.c @@ -129,11 +129,20 @@ DEFINE_VECTOR_TYPE (l1_dir, struct l1_entry); struct sparse_array { struct allocator a; /* Must come first. */ - /* This lock is highly contended. When hammering the memory plugin - * with 8 fio threads, about 30% of the total system time was taken - * just waiting for this lock. Fixing this is quite hard. + /* The shared (read) lock must be held if you just want to access + * the data without modifying any of the L1/L2 metadata or + * allocating or freeing any page. + * + * To modify the L1/L2 metadata including allocating or freeing any + * page you must hold the exclusive (write) lock. + * + * Because POSIX rwlocks are not upgradable this presents a problem. + * We solve it below by speculatively performing the request while + * holding the shared lock, but if we run into an operation that + * needs to update the metadata, we restart the entire request + * holding the exclusive lock. */ - pthread_mutex_t lock; + pthread_rwlock_t lock;