On Fri, Jun 14, 2024 at 09:15:34PM +0000, netpleb wrote:
Hi Rich and list,
Thanks for your reply and the suggestions. I kept doing some more tests, this time between two cloud servers in two parts of the world and was able to get the throughput up quite a bit.
I am not sure this was the exact culprit, but with my original tests, so as to avoid exposing the NBD tcp port to the entire world, I was doing it via SSH. However, when I removed ssh from the picture (temporarily, just as a test), then it seemed to allow much better throughput.
I wonder if using wireguard would be a way to avoid publicly exposing the nbd socket, but still retain most of the performance. Either that, or maybe just doing TLS like you suggested would be the way to go.
Although we do code nbdkit carefully, and regularly fuzz it, nbdkit and NBD in general are rarely exposed to the full force of the internet and so we don't know exactly what bugs or exploits are possible. Also simple denial of service attacks are possible (by design) if you allow anyone to connect, since they can amplify a tiny NBD request into returning almost arbitrarily large amounts of data. A firewall, or VPN tunnel like Wireguard, will prevent anyone unauthorized from connecting at all, which is an excellent extra layer of protection. These filters may be interesting: https://libguestfs.org/nbdkit-ip-filter.1.html https://libguestfs.org/nbdkit-limit-filter.1.html https://libguestfs.org/nbdkit-rate-filter.1.html https://libguestfs.org/nbdkit-exitlast-filter.1.html Using TLS with client certificates or TLS-PSK means that only the clients you choose will be able to finish negotiation and start reading and writing data. Using TLS only (without client certificates / PSK) does _not_ provide any protection from clients. It prevents a man-in-the-middle from sniffing or injecting data (which is a good thing), but that's all. Also there is a small part of the NBD protocol that happens in plaintext before the STARTTLS command which upgrades the connection to TLS. Even with nbdkit using TLS, any client can try to exploit that early part of the connection. A firewall, VPN or nbdkit-ip-filter protects against this. Plugins and filters might also be a source of exploits, and since there are so many of them, including custom plugins that you could write yourself, it's hard to quantify the risk there.
Sounds like in general I probably need to start using/testing with nbdkit, rather than nbd-server/client. I will look into that next. I was testing this with nixos on the server, and there is not yet a nixos module or package for nbdkit, so I was lazy and used nbd-server.
There are a good variety of NBD clients and servers now and they should all interoperate, as we test many of them routinely.
Anyway, the fact that it worked at all over a connection with 175ms of latency was really neat to see! From what I could tell, LUKs did not even cause much of a slowdown either.
Good luck! Rich.
On Friday, June 14th, 2024 at 10:12 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Mon, Jun 10, 2024 at 08:38:02PM +0000, netpleb wrote:
Hi Richard et. al,
I was attempting to ask this question via a comment on this post (https:// rwmj.wordpress.com/2022/05/14/nbdkit-now-supports-luks-encryption/) on your wordpress site, but it was giving me some trouble, so I will ask it here:
Sorry, at devconf.cz this week, so things are a little chaotic/busy ...
I hope this is not too naive of a question, but is nbdkit (or even just nbd in general) a good fit for mounting and using a LUKs encrypted block device over the internet?
It makes it possible.
Separate from the LUKS issue, whether it's a good idea to expose an NBD socket to the whole world is interesting. I would certainly be looking at TLS client certificates to try to control who can connect, and being careful to sandbox the machine running nbdkit.
https://libguestfs.org/nbdkit-tls.1.html
Specifically, the client would encrypt the block device using luks and thereby keep the luks secret key local like you did in your blog post.
I have done some rudimentary tests with nbd-client but have been unable to achieve speeds faster than approximately 1.4 MB/sec. However, doing the same thing on my local network gave me closer to 50 MB/sec. The latter seems usable, but the former seems not, and I would like to better understand if/how nbd can be used over the internet.
I'm going to guess that round trips and/or the client not using multi-conn are killing your performance. Multi-conn is really essential when trying to get good performance when you have large latencies. nbd-client -connections parameter should let you control it.
Another potential issue would be that the client is making lots of tiny requests and not coallescing. There are various filters including nbdkit-log-filter and nbdkit-stats-filter which let you view this.
You could also play with nbdcopy, since it allows you to control all the parameters, like multi-conn, requests in flight, request size, etc.
Rich.
Thank you in advance for your reply.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html