[Libguestfs] Re: client-side LUKs + NBD over internet?
On Fri, Jun 28, 2024 at 10:19:44PM -0000, jon.szymaniak.foss(a)gmail.com wrote:
Would you mind sharing a little bit of information about your setup?
Your local network setup appears to be performing much better than mine...
I am using LUKSv2 (not the nbdkit LUKSv1 plugin) atop of an nbdfuse mount point on the
client side.
nbdfuse goes through a lot of layers, and has had next to no attention
to optimization. It's very convenient, eg., for making quick changes
to disk images without requiring root permissions. But I think for
benchmarking you're better off using the kernel nbd.ko client. Make
sure to use nbd-client -connections 4 (or more) so that you're using
multi-conn.
Alternately fio has a libnbd client which is another good way to benchmark:
https://github.com/axboe/fio/blob/master/examples/nbd.fio#L1
Below are some high-level, non-scientific observations for write
speeds achieved while dd'ing /dev/zero to my target device from a client on a Gigabit
LAN.
nbdkit using 1G memory as the storage:
* 810 MiB/s - LUKS + nbdfuse + nbdkit, w/ TLS (Not a typo, weird outlier result)
Caching? Using fio + libnbd should eliminate any caching effects.
* ~15 MiB/s - nbdfuse + nbdkit, w/ TLS
* ~60 MiB/s - nbdfuse + nbdkit, no TLS
nbdkit with a SW RAID5 device (HDDs attached via USB3-scsi controller):
* ~1 MiB/s - LUKS + nbdfuse + nbdkit, w/ TLS
* ~12 MiB/s - nbdfuse + nbdkit, w/ TLS
* ~29 MiB/s - nbdfuse + nbdkit, no TLS
Server-side dd /dev/zero directly to SW RAID5 target (HDDs attached via USB3-scsi
controller):
* ~64 MiB/s
Both my client and server CPU and memory load seem quite
low...almost negligible. The disk activity lights on the backing
storage suggest to me that the disks are not being constantly
written to. The disk appear idle for long chunks of time, followed
by a short burst of activity. The disk write column in htop's IO tab
shows a bunch of nbdkit threads with values ~50 K/s.
I'm not surprised! nbdfuse spends all its time shuffling data through
multiple userspace & kernel layers and waiting on round trips.
Rich.
Any thoughts welcome... trying to decide where to start focusing
taking deeper measurements with and try some tuning.
Thanks,
Jon
_______________________________________________
Libguestfs mailing list -- guestfs(a)lists.libguestfs.org
To unsubscribe send an email to guestfs-leave(a)lists.libguestfs.org
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v