8:58 a.m.
On 9/12/19 12:41 PM, Richard W.M. Jones wrote:
We have discovered a potential Denial of Service / Amplification
Attack
in nbdkit.
Unfortunately, our fix for this issue cause another potential Denial of
Service attack:
Lifecycle
---------
Reported: 2019-09-11 Fixed: 2019-09-11 Published: 2019-09-12
There is no CVE number assigned for this issue yet, but the bug is
being categorized and processed by Red Hat's security team which may
result in a CVE being published later.
Reported: 2019-09-18 Fixed: 2019-09-19 Published: 2019-09-20
Also pending Red Hat security review for whether this deserves a CVE
(presumably either both issues, or neither, will have a CVE)
Credit
------
Reported and patched by Richard W.M. Jones <rjones(a)redhat.com>.
Reviewed by Eric Blake <eblake(a)redhat.com> and
Daniel Berrangé <berrange(a)redhat.com>.
Description
-----------
nbdkit is a Network Block Device (NBD) server with multiple plugins.
An attacker, by connecting to nbdkit’s TCP port, is able to make the
current plugin’s .open() callback run. For some plugins this callback
does a considerable amount of work, for example connecting to a remote
machine (nbdkit-ssh-plugin) or launching a small VM
(nbdkit-guestfs-plugin). Thus for a small amount of work done by the
attacker (opening many TCP connections but not sending any data), a
large amount of work is done by nbdkit (eg. making many full ssh
connections to a remote machine or launching hundreds of VMs).
A related problem is that if nbdkit is configured to use TLS for
client authentication, even clients which are not authorized are able
to attack nbdkit in this way because all they have to do is open a TCP
connection.
The NBD specification allows a client to request details about an export
(NBD_OPT_INFO) prior to committing to use that export (NBD_OPT_GO).
However, the fix to defer a plugin's .open() callback to as late as
possible means that a compliant client handshake now triggers
back-to-back opens from the same connection rather than sharing a single
open, and the second one fails due to an assertion because the first
open did not have a matching .close() callback. Most NBD clients do not
query info, therefore, a malicious client can use the assertion failure
by querying info to cause a denial of service to another client that
does not query info.
Mitigating this is that the current plugin is NOT selected by the
attacker, but is part of the server configuration. Therefore the
effectiveness of the attack depends on the particular configuration.
Mitigating this is that if TLS is used to filter out untrusted guest,
and you can ensure that trusted guests do not query info, then the
assertion is unreachable.
There are two situations in which you will NOT be affected:
* You are not exposing nbdkit over TCP (eg. using nbdkit -U), or the
TCP port is not accessible to untrusted attackers (eg. private LAN,
firewall rules).
* The plugin you are using does not do much work in its .open()
callback. For example the memory plugin does nothing in its
callback so is not vulnerable:
https://github.com/libguestfs/nbdkit/blob/afbe7f11098df256acf8eafa9d4ec1e...
Test if nbdkit is a vulnerable version
--------------------------------------
Run the following command (requires bash):
nbdkit -fv null --run 'sleep 1 >/dev/tcp/localhost/10809' 2>&1 |
grep -q 'open readonly' && echo vulnerable || echo patched
At this time, there are no known readily-available open source clients
which query info, thus no easy way to test for the second vulnerability
other than creating a custom client.
Workarounds
-----------
It is recommended to upgrade to a fixed version of nbdkit (see next
section). However if this cannot be done, apply network filtering
(eg. firewall, TCP wrappers, etc.) to ensure that untrusted clients
cannot connect to nbdkit’s public TCP port. Or use a Unix domain
socket instead of a loopback TCP connection.
Fixes
-----
This affects all versions of nbdkit. A fix is available for 1.12,
1.14 and the current development branch.
* development branch (1.15)
https://github.com/libguestfs/nbdkit/commit/c05686f9577fa91b6a3a4d8c06595...
or use nbdkit >= 1.15.1 from
http://download.libguestfs.org/nbdkit/1.15-development/
Use nbdkit >= 1.15.2, containing
https://github.com/libguestfs/nbdkit/commit/a6b88b195a959b17524d1c8353fd4...
* stable branch 1.14
https://github.com/libguestfs/nbdkit/commit/e06cde00659ff97182173d0e33fff...
or use nbdkit >= 1.14.1 from
http://download.libguestfs.org/nbdkit/1.14-stable/
Use nbdkit >= 1.14.2, containing
https://github.com/libguestfs/nbdkit/commit/bf0d61883a2f02f4388ec10dc92d4...
* stable branch 1.12
https://github.com/libguestfs/nbdkit/commit/22b30adb796bb6dca264a38598f80...
or use nbdkit >= 1.12.7 from
http://download.libguestfs.org/nbdkit/1.12-stable/
Use nbdkit >= 1.12.8, containing
https://github.com/libguestfs/nbdkit/commit/b2bc6683ea3cd1f6be694e8a681df...
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org