Hi Rich and list, Thanks for your reply and the suggestions. I kept doing some more tests, this time between two cloud servers in two parts of the world and was able to get the throughput up quite a bit. I am not sure this was the exact culprit, but with my original tests, so as to avoid exposing the NBD tcp port to the entire world, I was doing it via SSH. However, when I removed ssh from the picture (temporarily, just as a test), then it seemed to allow much better throughput. I wonder if using wireguard would be a way to avoid publicly exposing the nbd socket, but still retain most of the performance. Either that, or maybe just doing TLS like you suggested would be the way to go. Sounds like in general I probably need to start using/testing with nbdkit, rather than nbd-server/client. I will look into that next. I was testing this with nixos on the server, and there is not yet a nixos module or package for nbdkit, so I was lazy and used nbd-server. Anyway, the fact that it worked at all over a connection with 175ms of latency was really neat to see! From what I could tell, LUKs did not even cause much of a slowdown either. On Friday, June 14th, 2024 at 10:12 AM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: > On Mon, Jun 10, 2024 at 08:38:02PM +0000, netpleb wrote: > > > Hi Richard et. al, > > > > I was attempting to ask this question via a comment on this post (https:// > > rwmj.wordpress.com/2022/05/14/nbdkit-now-supports-luks-encryption/) on your > > wordpress site, but it was giving me some trouble, so I will ask it here: > > > Sorry, at devconf.cz this week, so things are a little chaotic/busy ... > > > I hope this is not too naive of a question, but is nbdkit (or even > > just nbd in general) a good fit for mounting and using a LUKs > > encrypted block device over the internet? > > > It makes it possible. > > Separate from the LUKS issue, whether it's a good idea to expose an > NBD socket to the whole world is interesting. I would certainly be > looking at TLS client certificates to try to control who can connect, > and being careful to sandbox the machine running nbdkit. > > https://libguestfs.org/nbdkit-tls.1.html > > > Specifically, the client would encrypt the block device using luks > > and thereby keep the luks secret key local like you did in your blog > > post. > > > > I have done some rudimentary tests with nbd-client but have been > > unable to achieve speeds faster than approximately 1.4 > > MB/sec. However, doing the same thing on my local network gave me > > closer to 50 MB/sec. The latter seems usable, but the former seems > > not, and I would like to better understand if/how nbd can be used > > over the internet. > > > I'm going to guess that round trips and/or the client not using > multi-conn are killing your performance. Multi-conn is really > essential when trying to get good performance when you have large > latencies. nbd-client -connections parameter should let you control > it. > > Another potential issue would be that the client is making lots of > tiny requests and not coallescing. There are various filters > including nbdkit-log-filter and nbdkit-stats-filter which let you view > this. > > You could also play with nbdcopy, since it allows you to control all > the parameters, like multi-conn, requests in flight, request size, > etc. > > Rich. > > > Thank you in advance for your reply. > > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-builder quickly builds VMs from scratch > http://libguestfs.org/virt-builder.1.html

Would you mind sharing a little bit of information about your setup? Your local network setup appears to be performing much better than mine... I am using LUKSv2 (not the nbdkit LUKSv1 plugin) atop of an nbdfuse mount point on the client side. Below are some high-level, non-scientific observations for write speeds achieved while dd'ing /dev/zero to my target device from a client on a Gigabit LAN. nbdkit using 1G memory as the storage: * 810 MiB/s - LUKS + nbdfuse + nbdkit, w/ TLS (Not a typo, weird outlier result) * ~15 MiB/s - nbdfuse + nbdkit, w/ TLS * ~60 MiB/s - nbdfuse + nbdkit, no TLS nbdkit with a SW RAID5 device (HDDs attached via USB3-scsi controller): * ~1 MiB/s - LUKS + nbdfuse + nbdkit, w/ TLS * ~12 MiB/s - nbdfuse + nbdkit, w/ TLS * ~29 MiB/s - nbdfuse + nbdkit, no TLS Server-side dd /dev/zero directly to SW RAID5 target (HDDs attached via USB3-scsi controller): * ~64 MiB/s Both my client and server CPU and memory load seem quite low...almost negligible. The disk activity lights on the backing storage suggest to me that the disks are not being constantly written to. The disk appear idle for long chunks of time, followed by a short burst of activity. The disk write column in htop's IO tab shows a bunch of nbdkit threads with values ~50 K/s. Any thoughts welcome... trying to decide where to start focusing taking deeper measurements with and try some tuning. Thanks, Jon