client-side LUKs + NBD over internet?
Hi Richard et. al, I was attempting to ask this question via a comment on this post (https://rwmj.wordpress.com/2022/05/14/nbdkit-now-supports-luks-encryption/) on your wordpress site, but it was giving me some trouble, so I will ask it here: I hope this is not too naive of a question, but is nbdkit (or even just nbd in general) a good fit for mounting and using a LUKs encrypted block device over the internet? Specifically, the client would encrypt the block device using luks and thereby keep the luks secret key local like you did in your blog post. I have done some rudimentary tests with nbd-client but have been unable to achieve speeds faster than approximately 1.4 MB/sec. However, doing the same thing on my local network gave me closer to 50 MB/sec. The latter seems usable, but the former seems not, and I would like to better understand if/how nbd can be used over the internet. Thank you in advance for your reply.
On Mon, Jun 10, 2024 at 08:38:02PM +0000, netpleb wrote:
Hi Richard et. al,
I was attempting to ask this question via a comment on this post (https:// rwmj.wordpress.com/2022/05/14/nbdkit-now-supports-luks-encryption/) on your wordpress site, but it was giving me some trouble, so I will ask it here:
Sorry, at devconf.cz this week, so things are a little chaotic/busy ...
I hope this is not too naive of a question, but is nbdkit (or even just nbd in general) a good fit for mounting and using a LUKs encrypted block device over the internet?
It makes it possible. Separate from the LUKS issue, whether it's a good idea to expose an NBD socket to the whole world is interesting. I would certainly be looking at TLS client certificates to try to control who can connect, and being careful to sandbox the machine running nbdkit. https://libguestfs.org/nbdkit-tls.1.html
Specifically, the client would encrypt the block device using luks and thereby keep the luks secret key local like you did in your blog post.
I have done some rudimentary tests with nbd-client but have been unable to achieve speeds faster than approximately 1.4 MB/sec. However, doing the same thing on my local network gave me closer to 50 MB/sec. The latter seems usable, but the former seems not, and I would like to better understand if/how nbd can be used over the internet.
I'm going to guess that round trips and/or the client not using multi-conn are killing your performance. Multi-conn is really essential when trying to get good performance when you have large latencies. nbd-client -connections parameter should let you control it. Another potential issue would be that the client is making lots of tiny requests and not coallescing. There are various filters including nbdkit-log-filter and nbdkit-stats-filter which let you view this. You could also play with nbdcopy, since it allows you to control all the parameters, like multi-conn, requests in flight, request size, etc. Rich.
Thank you in advance for your reply.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Hi Rich and list, Thanks for your reply and the suggestions. I kept doing some more tests, this time between two cloud servers in two parts of the world and was able to get the throughput up quite a bit. I am not sure this was the exact culprit, but with my original tests, so as to avoid exposing the NBD tcp port to the entire world, I was doing it via SSH. However, when I removed ssh from the picture (temporarily, just as a test), then it seemed to allow much better throughput. I wonder if using wireguard would be a way to avoid publicly exposing the nbd socket, but still retain most of the performance. Either that, or maybe just doing TLS like you suggested would be the way to go. Sounds like in general I probably need to start using/testing with nbdkit, rather than nbd-server/client. I will look into that next. I was testing this with nixos on the server, and there is not yet a nixos module or package for nbdkit, so I was lazy and used nbd-server. Anyway, the fact that it worked at all over a connection with 175ms of latency was really neat to see! From what I could tell, LUKs did not even cause much of a slowdown either. On Friday, June 14th, 2024 at 10:12 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Mon, Jun 10, 2024 at 08:38:02PM +0000, netpleb wrote:
Hi Richard et. al,
I was attempting to ask this question via a comment on this post (https:// rwmj.wordpress.com/2022/05/14/nbdkit-now-supports-luks-encryption/) on your wordpress site, but it was giving me some trouble, so I will ask it here:
Sorry, at devconf.cz this week, so things are a little chaotic/busy ...
I hope this is not too naive of a question, but is nbdkit (or even just nbd in general) a good fit for mounting and using a LUKs encrypted block device over the internet?
It makes it possible.
Separate from the LUKS issue, whether it's a good idea to expose an NBD socket to the whole world is interesting. I would certainly be looking at TLS client certificates to try to control who can connect, and being careful to sandbox the machine running nbdkit.
https://libguestfs.org/nbdkit-tls.1.html
Specifically, the client would encrypt the block device using luks and thereby keep the luks secret key local like you did in your blog post.
I have done some rudimentary tests with nbd-client but have been unable to achieve speeds faster than approximately 1.4 MB/sec. However, doing the same thing on my local network gave me closer to 50 MB/sec. The latter seems usable, but the former seems not, and I would like to better understand if/how nbd can be used over the internet.
I'm going to guess that round trips and/or the client not using multi-conn are killing your performance. Multi-conn is really essential when trying to get good performance when you have large latencies. nbd-client -connections parameter should let you control it.
Another potential issue would be that the client is making lots of tiny requests and not coallescing. There are various filters including nbdkit-log-filter and nbdkit-stats-filter which let you view this.
You could also play with nbdcopy, since it allows you to control all the parameters, like multi-conn, requests in flight, request size, etc.
Rich.
Thank you in advance for your reply.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Fri, Jun 14, 2024 at 09:15:34PM +0000, netpleb wrote:
Hi Rich and list,
Thanks for your reply and the suggestions. I kept doing some more tests, this time between two cloud servers in two parts of the world and was able to get the throughput up quite a bit.
I am not sure this was the exact culprit, but with my original tests, so as to avoid exposing the NBD tcp port to the entire world, I was doing it via SSH. However, when I removed ssh from the picture (temporarily, just as a test), then it seemed to allow much better throughput.
I wonder if using wireguard would be a way to avoid publicly exposing the nbd socket, but still retain most of the performance. Either that, or maybe just doing TLS like you suggested would be the way to go.
Although we do code nbdkit carefully, and regularly fuzz it, nbdkit and NBD in general are rarely exposed to the full force of the internet and so we don't know exactly what bugs or exploits are possible. Also simple denial of service attacks are possible (by design) if you allow anyone to connect, since they can amplify a tiny NBD request into returning almost arbitrarily large amounts of data. A firewall, or VPN tunnel like Wireguard, will prevent anyone unauthorized from connecting at all, which is an excellent extra layer of protection. These filters may be interesting: https://libguestfs.org/nbdkit-ip-filter.1.html https://libguestfs.org/nbdkit-limit-filter.1.html https://libguestfs.org/nbdkit-rate-filter.1.html https://libguestfs.org/nbdkit-exitlast-filter.1.html Using TLS with client certificates or TLS-PSK means that only the clients you choose will be able to finish negotiation and start reading and writing data. Using TLS only (without client certificates / PSK) does _not_ provide any protection from clients. It prevents a man-in-the-middle from sniffing or injecting data (which is a good thing), but that's all. Also there is a small part of the NBD protocol that happens in plaintext before the STARTTLS command which upgrades the connection to TLS. Even with nbdkit using TLS, any client can try to exploit that early part of the connection. A firewall, VPN or nbdkit-ip-filter protects against this. Plugins and filters might also be a source of exploits, and since there are so many of them, including custom plugins that you could write yourself, it's hard to quantify the risk there.
Sounds like in general I probably need to start using/testing with nbdkit, rather than nbd-server/client. I will look into that next. I was testing this with nixos on the server, and there is not yet a nixos module or package for nbdkit, so I was lazy and used nbd-server.
There are a good variety of NBD clients and servers now and they should all interoperate, as we test many of them routinely.
Anyway, the fact that it worked at all over a connection with 175ms of latency was really neat to see! From what I could tell, LUKs did not even cause much of a slowdown either.
Good luck! Rich.
On Friday, June 14th, 2024 at 10:12 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Mon, Jun 10, 2024 at 08:38:02PM +0000, netpleb wrote:
Hi Richard et. al,
I was attempting to ask this question via a comment on this post (https:// rwmj.wordpress.com/2022/05/14/nbdkit-now-supports-luks-encryption/) on your wordpress site, but it was giving me some trouble, so I will ask it here:
Sorry, at devconf.cz this week, so things are a little chaotic/busy ...
I hope this is not too naive of a question, but is nbdkit (or even just nbd in general) a good fit for mounting and using a LUKs encrypted block device over the internet?
It makes it possible.
Separate from the LUKS issue, whether it's a good idea to expose an NBD socket to the whole world is interesting. I would certainly be looking at TLS client certificates to try to control who can connect, and being careful to sandbox the machine running nbdkit.
https://libguestfs.org/nbdkit-tls.1.html
Specifically, the client would encrypt the block device using luks and thereby keep the luks secret key local like you did in your blog post.
I have done some rudimentary tests with nbd-client but have been unable to achieve speeds faster than approximately 1.4 MB/sec. However, doing the same thing on my local network gave me closer to 50 MB/sec. The latter seems usable, but the former seems not, and I would like to better understand if/how nbd can be used over the internet.
I'm going to guess that round trips and/or the client not using multi-conn are killing your performance. Multi-conn is really essential when trying to get good performance when you have large latencies. nbd-client -connections parameter should let you control it.
Another potential issue would be that the client is making lots of tiny requests and not coallescing. There are various filters including nbdkit-log-filter and nbdkit-stats-filter which let you view this.
You could also play with nbdcopy, since it allows you to control all the parameters, like multi-conn, requests in flight, request size, etc.
Rich.
Thank you in advance for your reply.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Would you mind sharing a little bit of information about your setup? Your local network setup appears to be performing much better than mine... I am using LUKSv2 (not the nbdkit LUKSv1 plugin) atop of an nbdfuse mount point on the client side. Below are some high-level, non-scientific observations for write speeds achieved while dd'ing /dev/zero to my target device from a client on a Gigabit LAN. nbdkit using 1G memory as the storage: * 810 MiB/s - LUKS + nbdfuse + nbdkit, w/ TLS (Not a typo, weird outlier result) * ~15 MiB/s - nbdfuse + nbdkit, w/ TLS * ~60 MiB/s - nbdfuse + nbdkit, no TLS nbdkit with a SW RAID5 device (HDDs attached via USB3-scsi controller): * ~1 MiB/s - LUKS + nbdfuse + nbdkit, w/ TLS * ~12 MiB/s - nbdfuse + nbdkit, w/ TLS * ~29 MiB/s - nbdfuse + nbdkit, no TLS Server-side dd /dev/zero directly to SW RAID5 target (HDDs attached via USB3-scsi controller): * ~64 MiB/s Both my client and server CPU and memory load seem quite low...almost negligible. The disk activity lights on the backing storage suggest to me that the disks are not being constantly written to. The disk appear idle for long chunks of time, followed by a short burst of activity. The disk write column in htop's IO tab shows a bunch of nbdkit threads with values ~50 K/s. Any thoughts welcome... trying to decide where to start focusing taking deeper measurements with and try some tuning. Thanks, Jon
On Fri, Jun 28, 2024 at 10:19:44PM -0000, jon.szymaniak.foss@gmail.com wrote:
Would you mind sharing a little bit of information about your setup? Your local network setup appears to be performing much better than mine...
I am using LUKSv2 (not the nbdkit LUKSv1 plugin) atop of an nbdfuse mount point on the client side.
nbdfuse goes through a lot of layers, and has had next to no attention to optimization. It's very convenient, eg., for making quick changes to disk images without requiring root permissions. But I think for benchmarking you're better off using the kernel nbd.ko client. Make sure to use nbd-client -connections 4 (or more) so that you're using multi-conn. Alternately fio has a libnbd client which is another good way to benchmark: https://github.com/axboe/fio/blob/master/examples/nbd.fio#L1
Below are some high-level, non-scientific observations for write speeds achieved while dd'ing /dev/zero to my target device from a client on a Gigabit LAN.
nbdkit using 1G memory as the storage: * 810 MiB/s - LUKS + nbdfuse + nbdkit, w/ TLS (Not a typo, weird outlier result)
Caching? Using fio + libnbd should eliminate any caching effects.
* ~15 MiB/s - nbdfuse + nbdkit, w/ TLS * ~60 MiB/s - nbdfuse + nbdkit, no TLS
nbdkit with a SW RAID5 device (HDDs attached via USB3-scsi controller): * ~1 MiB/s - LUKS + nbdfuse + nbdkit, w/ TLS * ~12 MiB/s - nbdfuse + nbdkit, w/ TLS * ~29 MiB/s - nbdfuse + nbdkit, no TLS
Server-side dd /dev/zero directly to SW RAID5 target (HDDs attached via USB3-scsi controller): * ~64 MiB/s
Both my client and server CPU and memory load seem quite low...almost negligible. The disk activity lights on the backing storage suggest to me that the disks are not being constantly written to. The disk appear idle for long chunks of time, followed by a short burst of activity. The disk write column in htop's IO tab shows a bunch of nbdkit threads with values ~50 K/s.
I'm not surprised! nbdfuse spends all its time shuffling data through multiple userspace & kernel layers and waiting on round trips. Rich.
Any thoughts welcome... trying to decide where to start focusing taking deeper measurements with and try some tuning.
Thanks, Jon _______________________________________________ Libguestfs mailing list -- guestfs@lists.libguestfs.org To unsubscribe send an email to guestfs-leave@lists.libguestfs.org
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
participants (3)
-
jon.szymaniak.foss@gmail.com -
netpleb -
Richard W.M. Jones