alternatives for hooking dlopen() without LD_LIBRARY_PATH or LD_AUDIT?
I've got a situation where I need to hook a dlopen() made by VDDK, a proprietary library, where it passes a relative name expecting to resolve to a copy of several libraries, including libstdc++.so, that it installs alongside itself, and fails to load if that resolves to the system libstdc++.so. The simplest solution of providing LD_LIBRARY_PATH is enough to load VDDK, but then poisons any child process which likewise fail to load if they pick up VDDK's libstdc++.so instead of the system one. Up to now, we've documented throwing the burden on the end user who has to write convoluted: LD_LIBRARY_PATH_save=$LD_LIBRARY_PATH \ LD_LIBRARY_PATH=/path/to/vddklibs:$LD_LIBRARY_PATH \ nbdkit vddk libdir=/path/to/vddklibs file --run \ 'LD_LIBRARY_PATH=$LD_LIBRARY_PATH_save; program args' where we would rather the end-user could get away with a more concise: nbdkit vddk libdir=/path/to/vddklibs file --run 'program args' Sequentially, we have this scenario: nbdkit vddk libdir=/path/to/libs file --run 'program args' - nbdkit binary calls dlopen("/path/to/nbdkit-vddk-plugin.so") - nbdkit-vddk-plugin.so calls dlopen("/path/to/libs/libvixDiskLibs.so") using the libdir= argument to load vddk (rather than dlopen("libvixDiskLibs.so") relying on LD_LIBRARY_PATH) - vddk's initializer calls dlopen("libcrypto.so") expecting to open /path/to/libs/libcrypto.so, but either LD_LIBRARY_PATH made that possible (at which point we have to scrub it before a child process will be penalized), or we have to find a way to rewrite vddk's dlopen call from relative into absolute before passing it to the real dlopen - nbdkit binary spawns a child process to exec 'program args' - program does not want /path/to/libs in its search path Writing my own dlopen() wrapper directly in nbdkit seems like a non-starter (my override has to come from a shared library before it can replace the shared version that would be imported from -ldl, at least for all subsequent shared library loads that want to bind to the override). And if I read 'man dlopen' correctly, since nbdkit used dlopen() to load nbdkit-vddk-plugin.so, then dlopen() is already bound in the main context, so unless I use RTLD_DEEPBIND from nbdkit, then nbdkit-vddk-plugin.so will also see dlopen() bound to -dl rather than anything it loads locally; but even with RTLD_DEEPBIND, it sounds like that higher precedence lasts only for nbdkit-vddk-plugin.so and does not extend to later bindings performed for libvixDiskLib.so (which means vddk is back to -ldl's dlopen, without my hook). Thus, to hook dlopen within the same process, I need some way to create a scope where I can provide a shared dlopen() that will take precedence when resolving symbols during the load of libvixDiskLib.so, but where that hook code can still defer back to the real dlopen() from -ldl and does not penalize child processes. I managed to create a solution that avoids the need to set LD_PRELOAD_PATH at all by installing a shared library that hooks dlopen(), then loading both my shim and vddk via dlmopen() without the use of RTLD_GLOBAL or RTLD_DEEPBIND. More links on my solution: https://www.redhat.com/archives/libguestfs/2020-February/msg00154.html https://bugzilla.redhat.com/show_bug.cgi?id=1756307#c7 https://sourceware.org/bugzilla/show_bug.cgi?id=15971#c5 However, when Florian saw it, he suggested that my solution of dlmopen() for a shim library that overrides dlopen() is reinventing what la_objsearch() can already do. This is in part because the moment you dlmopen() a library into a separate namespace, you can't debug it (both glibc and gdb need additional patches to expose alternative namespaces for debugging), but there may be other nasty surprises lurking. But after spending more than an hour playing with la_objsearch() and reading 'man rtld-audit', it looks like an audit library cannot be triggered in glibc except by listing it in LD_AUDIT in the environment during exec - which is back to the same problem we have with needing LD_LIBRARY_PATH in the environment. Furthermore, although I know that glibc's audit interface is slightly different from the Solaris version it copied from, the Solaris documentation states that an audit library has some rather tough restrictions (including that using 'malloc' is unsafe, https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-3.html#scrolltoc "Some system interfaces assume that the interfaces are the only instance of their implementation within a process. Examples of such implementations are signals and malloc(3C). Audit libraries should avoid using such interfaces, as doing so can inadvertently alter the behavior of the application."). But Solaris also stated that a library could serve as an audit entry point without LD_AUDIT if it is registered locally, via -Wl,-paudit.so.1 when creating the shared library (https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-18.html#scrolltoc); it doesn't seem that this functionality exists with glibc (/usr/lib64/libaudit.so on Linux has nothing to do with rtld-audit). Does anyone have any ideas on how to let a shared library implement an audit interface for just its own process, without having to edit LD_AUDIT or re-exec the process? Or is there yet another way to hook a program to rewrite misbehaving dlopen() calls without relying on either dlmopen() or la_objsearch(), or requiring pre-set environment variables, or having to re-exec a process? -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On 2/14/20 1:02 PM, Eric Blake wrote:
Writing my own dlopen() wrapper directly in nbdkit seems like a non-starter (my override has to come from a shared library before it can replace the shared version that would be imported from -ldl, at least for all subsequent shared library loads that want to bind to the override).
Maybe I spoke too soon. I've tried another approach that looks like it will do what I want: put my shim dlopen() in a shared library, but link nbdkit against that shared library PRIOR to -ldl (so that name lookup for dlopen resolves there first). The shim library in turn depends on -ldl so that dlsym(RTLD_NEXT, "dlopen") still lets me get to the real dlopen. And by linking it directly into nbdkit, rather than into the nbdkit-vddk-plugin.so that gets loaded later, the first bound dlopen() in use for all subsequent loads is from my shim. It's still a bit less clean than I'd like (it requires tighter coupling between nbdkit and nbdkit-vddk-plugin.so than what used to exist), but the fact that it works without dlmopen() or LD_LIBRARY_PATH is in its favor. I'm now polishing up the experiment, and will post the patch when it's ready. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On 2/14/20 4:29 PM, Eric Blake wrote:
On 2/14/20 1:02 PM, Eric Blake wrote:
Writing my own dlopen() wrapper directly in nbdkit seems like a non-starter (my override has to come from a shared library before it can replace the shared version that would be imported from -ldl, at least for all subsequent shared library loads that want to bind to the override).
Maybe I spoke too soon. I've tried another approach that looks like it will do what I want: put my shim dlopen() in a shared library, but link nbdkit against that shared library PRIOR to -ldl (so that name lookup for dlopen resolves there first). The shim library in turn depends on -ldl so that dlsym(RTLD_NEXT, "dlopen") still lets me get to the real dlopen. And by linking it directly into nbdkit, rather than into the nbdkit-vddk-plugin.so that gets loaded later, the first bound dlopen() in use for all subsequent loads is from my shim. It's still a bit less clean than I'd like (it requires tighter coupling between nbdkit and nbdkit-vddk-plugin.so than what used to exist), but the fact that it works without dlmopen() or LD_LIBRARY_PATH is in its favor. I'm now polishing up the experiment, and will post the patch when it's ready.
I think that's the best solution you're going to get. The alternatives (LD_LIBRARY_PATH, direct loader invocation, dlmopen) all have limitations that aren't helpful to your particular design. You have design strategies that look like this: - Move the object higher in the search order at link time (interposition) - Investigate static link order. - Investigate dynamic loader search order - Change what object is searched for - LD_LIBRARY_PATH, DT_RPATH, DT_RUNPATH, etc. - LD_AUDIT with la_objsearch() Your "shim dlopen()" is a case of moving the static link order around to ensure your shim is used first. -- Cheers, Carlos.
On 2/14/20 3:29 PM, Eric Blake wrote:
On 2/14/20 1:02 PM, Eric Blake wrote:
Writing my own dlopen() wrapper directly in nbdkit seems like a non-starter (my override has to come from a shared library before it can replace the shared version that would be imported from -ldl, at least for all subsequent shared library loads that want to bind to the override).
Maybe I spoke too soon. I've tried another approach that looks like it will do what I want: put my shim dlopen() in a shared library, but link nbdkit against that shared library PRIOR to -ldl (so that name lookup for dlopen resolves there first). The shim library in turn depends on -ldl so that dlsym(RTLD_NEXT, "dlopen") still lets me get to the real dlopen. And by linking it directly into nbdkit, rather than into the nbdkit-vddk-plugin.so that gets loaded later, the first bound dlopen() in use for all subsequent loads is from my shim. It's still a bit less clean than I'd like (it requires tighter coupling between nbdkit and nbdkit-vddk-plugin.so than what used to exist), but the fact that it works without dlmopen() or LD_LIBRARY_PATH is in its favor. I'm now polishing up the experiment, and will post the patch when it's ready.
Progress report: I've posted a v4 series that relies on a shared library in the main executable; but I'm still trying to see if I can further reduce things (maybe with -rdynamic) so that the main binary itself provides the dlopen() override without needing an auxiliary shared library. https://www.redhat.com/archives/libguestfs/2020-February/msg00162.html
But after spending more than an hour playing with la_objsearch() and reading 'man rtld-audit', it looks like an audit library cannot be triggered in glibc except by listing it in LD_AUDIT in the environment during exec - which is back to the same problem we have with needing LD_LIBRARY_PATH in the environment. Furthermore, although I know that glibc's audit interface is slightly different from the Solaris version it copied from, the Solaris documentation states that an audit library has some rather tough restrictions (including that using 'malloc' is unsafe, https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-3.html#scrolltoc "Some system interfaces assume that the interfaces are the only instance of their implementation within a process. Examples of such implementations are signals and malloc(3C). Audit libraries should avoid using such interfaces, as doing so can inadvertently alter the behavior of the application."). But Solaris also stated that a library could serve as an audit entry point without LD_AUDIT if it is registered locally, via -Wl,-paudit.so.1 when creating the shared library (https://docs.oracle.com/cd/E36784_01/html/E36857/chapter6-18.html#scrolltoc); it doesn't seem that this functionality exists with glibc (/usr/lib64/libaudit.so on Linux has nothing to do with rtld-audit).
I'm just now noticing that 'man ld' reports that you may pass '--audit LIB' during linking to add a DT_DEPAUDIT dependency on a library implementing the audit interface, which sounds like it might be an alternative to LD_AUDIT for getting a library with la_objsearch() to actually do something (but doesn't obviate the need for la_obsearch() to be in a separate library, rather than part of the main executable, unless a library can be reused as its own audit library...). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
* Eric Blake:
I'm just now noticing that 'man ld' reports that you may pass '--audit LIB' during linking to add a DT_DEPAUDIT dependency on a library implementing the audit interface, which sounds like it might be an alternative to LD_AUDIT for getting a library with la_objsearch() to actually do something (but doesn't obviate the need for la_obsearch() to be in a separate library, rather than part of the main executable, unless a library can be reused as its own audit library...).
DT_AUDIT support has yet to be implemented in glibc: <https://sourceware.org/bugzilla/show_bug.cgi?id=24943> <https://sourceware.org/ml/libc-alpha/2019-08/msg00705.html> If you go on record saying that you need this, maybe someone will review the patch. Sorry. 8-( Florian
On 2/17/20 9:12 AM, Florian Weimer wrote:
* Eric Blake:
I'm just now noticing that 'man ld' reports that you may pass '--audit LIB' during linking to add a DT_DEPAUDIT dependency on a library implementing the audit interface, which sounds like it might be an alternative to LD_AUDIT for getting a library with la_objsearch() to actually do something (but doesn't obviate the need for la_obsearch() to be in a separate library, rather than part of the main executable, unless a library can be reused as its own audit library...).
DT_AUDIT support has yet to be implemented in glibc:
<https://sourceware.org/bugzilla/show_bug.cgi?id=24943> <https://sourceware.org/ml/libc-alpha/2019-08/msg00705.html>
If you go on record saying that you need this, maybe someone will review the patch. Sorry. 8-(
Another followup: nbdkit-vddk-plugin.so is now using a re-exec setup [1], for several reasons: 1. all our other ideas tried so far (such as a dlopen() interposition in the main nbdkit binary) touched more files and required more exported APIs; we managed to get re-exec working with changes limited to just the plugin (other than a minor change to nbdkit to quit messing with argv[] so that /proc/self/cmdline would stay stable - but that did not require a new API). 2. it turns out that overriding dlopen() was insufficient to work around VDDK's setup [2]. It _did_ solve the initial dlopen() performed by VDDK, but that library in turn had DT_NEEDED entries for bare library names, which dlopen() does NOT affect but which la_objsearch() should. 3. for nbdkit, we want to minimize the number of binary files shipped; the re-exec solution works with just the nbdkit binary and the nbdkit-vddk-plugin.so. Any solution that requires a third file to be shipped (be that a shared library providing dlopen, or a LD_AUDIT library, or otherwise) is less palatable than the 2-binary solution that our re-exec solution provides. [1] https://github.com/libguestfs/nbdkit/commit/0c7ac4e655b [2] https://www.redhat.com/archives/libguestfs/2020-February/msg00184.html So with that said, here's a question I just thought of: If your patch for glibc support for DT_AUDIT is incorporated, is it possible to mark a shared library as its own audit library via DT_AUDIT? That is, if nbdkit-vddk-plugin.so can provide entry points for _both_ the nbdkit interface (which satisfies dlopen() from the nbdkit binary) and la_version/la_objsearch() (which satisfy the requirements for use from the audit code in ld.so), _and_ during the compilation of nbdkit-vddk-plugin.so, we marked the library as its own DT_AUDIT entry, would the mere act of dlopen("nbdkit-vddk-plugin.so") from nbdkit be sufficient to trigger audit actions such as la_objsearch() on all subsequent shared loads (whether by dlopen() or DT_NEEDED) performed by nbdkit-vddk-plugin.so and its descendant loaded libraries? Because if so, we would have a use case where a single binary, set up to act as its own audit library, might be sufficient to hook the shared object search path without needing any of environment variable modification, a process re-exec, or a third shipping binary - in which case that would indeed be a nicer solution than the current re-exec solution we committed today (of course, nbdkit would not be able to rely on that solution except on systems with new enough glibc to support DT_AUDIT). I guess even without DT_AUDIT support, I could at least answer the question of whether a single .so can be used to satisfy both dlopen() and LD_AUDIT interfaces at once by setting LD_AUDIT (where the only remaining gap is figuring out when glibc can let DT_AUDIT have the same effect). During my earlier attempts to get a working dlopen() override, I didn't consider any solution that required setting LD_AUDIT, but now that I proved dlopen() override alone was not enough for the case at hand, having to re-exec to set LD_AUDIT is no worse than having to re-exec to set LD_LIBRARY_PATH as a fallback to systems where glibc does not support DT_AUDIT. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
* Eric Blake:
So with that said, here's a question I just thought of:
If your patch for glibc support for DT_AUDIT is incorporated, is it possible to mark a shared library as its own audit library via DT_AUDIT? That is, if nbdkit-vddk-plugin.so can provide entry points for _both_ the nbdkit interface (which satisfies dlopen() from the nbdkit binary) and la_version/la_objsearch() (which satisfy the requirements for use from the audit code in ld.so), _and_ during the compilation of nbdkit-vddk-plugin.so, we marked the library as its own DT_AUDIT entry, would the mere act of dlopen("nbdkit-vddk-plugin.so") from nbdkit be sufficient to trigger audit actions such as la_objsearch() on all subsequent shared loads (whether by dlopen() or DT_NEEDED) performed by nbdkit-vddk-plugin.so and its descendant loaded libraries?
So you want to dlopen nbdkit-vddk-plugin.so and launch a new auditor even if the process so far hasn't used auditing? And the main program (which links agains a library which eventually makes this dlopen call) would not know anything about the existence of this specific plugin and auditing? This isn't currently supported. It's not just that the glibc implementation cannot do it. The audit API (as sketched in <link.h>) is not a good fit for late loading where you have never observed open events. It pretty much assumes that auditors are loaded magically *before* program start, so that they can observe all open calls and set up their own data structures along the way. I think what confuses me is that keep talking about a single binary, but clearly there is this separate vddk DSO, and there is talk of plugins. So it seems to me that multiple files are involved already? Thanks, Florian
On 2/21/20 6:19 AM, Florian Weimer wrote:
* Eric Blake:
So with that said, here's a question I just thought of:
If your patch for glibc support for DT_AUDIT is incorporated, is it possible to mark a shared library as its own audit library via DT_AUDIT? That is, if nbdkit-vddk-plugin.so can provide entry points for _both_ the nbdkit interface (which satisfies dlopen() from the nbdkit binary) and la_version/la_objsearch() (which satisfy the requirements for use from the audit code in ld.so), _and_ during the compilation of nbdkit-vddk-plugin.so, we marked the library as its own DT_AUDIT entry, would the mere act of dlopen("nbdkit-vddk-plugin.so") from nbdkit be sufficient to trigger audit actions such as la_objsearch() on all subsequent shared loads (whether by dlopen() or DT_NEEDED) performed by nbdkit-vddk-plugin.so and its descendant loaded libraries?
So you want to dlopen nbdkit-vddk-plugin.so and launch a new auditor even if the process so far hasn't used auditing? And the main program (which links agains a library which eventually makes this dlopen call) would not know anything about the existence of this specific plugin and auditing?
Yes, you interpreted my question correctly.
This isn't currently supported. It's not just that the glibc implementation cannot do it. The audit API (as sketched in <link.h>) is not a good fit for late loading where you have never observed open events. It pretty much assumes that auditors are loaded magically *before* program start, so that they can observe all open calls and set up their own data structures along the way.
The concern is not about nbdkit loading nbdkit-vddk-plugin.so, but nbdkit-vddk-plugin.so doing subsequent loads of libvixDiskLib.so and its bare dependencies on libstdc++.so and such that were incorrectly built without DT_RUNPATH, but where we can't rewrite libvixDiskLib.so because it is proprietary, so the best we can do is hook the loading environment (either by la_objsearch or by re-exec with LD_LIBRARY_PATH).
I think what confuses me is that keep talking about a single binary, but clearly there is this separate vddk DSO, and there is talk of plugins. So it seems to me that multiple files are involved already?
You are correct that there are multiple files involved: The nbdkit project currently has 2 relevant files: 'nbdkit' and 'nbdkit-vddk-plugin.so' (and various other plugins, but those are not relevant to the VDDK use case) The VDDK project from VMware: multiple files: libvixDiskLib.so (primary interface), which dlopen()s libdiskLibPlugin.so, which in turn has DT_NEEDED on libstdc++.so and several other recompiled system libraries. 'find vmware-vix-disklib-distrib/lib64/ -type f | wc' shows 23 libraries total, but the end user installs it as a single tarball from VMware. We can't change what VDDK ships, but we want to avoid making the nbdkit portion change from 2 files into 3, as every additional file required beyond what VMware ships is that much more burden for a user to choose to use nbdkit for accessing their VMware disks. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
I think what confuses me is that keep talking about a single binary, but clearly there is this separate vddk DSO, and there is talk of plugins. So it seems to me that multiple files are involved already?
nbdkit is a standalone binary that happens to be able to load plugins from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the path for plugins, and there's a wrapper allowing it to get local plugins even when it's still in the build directory. Adding another file would mean another path (or overloading the meaning of the plugin path) and just makes the whole thing more fragile and complex. Having said all that, what would also solve this is either an API for updating LD_LIBRARY_PATH after the program has started; or making setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant which takes a library path as an extra parameter. Rich. * “Why does setenv ("LD_LIBRARY_PATH") not work?” has several stackoverflow answers. Apparently even the JDK has to work around this by re-execing. https://www.google.com/search?q=setenv+%22LD_LIBRARY_PATH%22+site:stackoverf... -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
* Richard W. M. Jones:
On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
I think what confuses me is that keep talking about a single binary, but clearly there is this separate vddk DSO, and there is talk of plugins. So it seems to me that multiple files are involved already?
nbdkit is a standalone binary that happens to be able to load plugins from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the path for plugins, and there's a wrapper allowing it to get local plugins even when it's still in the build directory. Adding another file would mean another path (or overloading the meaning of the plugin path) and just makes the whole thing more fragile and complex.
Having said all that, what would also solve this is either an API for updating LD_LIBRARY_PATH after the program has started; or making setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant which takes a library path as an extra parameter.
Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so? Or does the path have to be chosen dynamically? If you merely want to prevent loading of libstdc++.so or libcrypto.so by vddk, it may be possible to explicitly dlopen DSOs of that name before loading vddk. But there is an existing bug where we do not duplicate properly on soname alone, so we may have to fix that first. Thanks, Florian
On Fri, Feb 21, 2020 at 04:00:30PM +0100, Florian Weimer wrote:
* Richard W. M. Jones:
On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
I think what confuses me is that keep talking about a single binary, but clearly there is this separate vddk DSO, and there is talk of plugins. So it seems to me that multiple files are involved already?
nbdkit is a standalone binary that happens to be able to load plugins from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the path for plugins, and there's a wrapper allowing it to get local plugins even when it's still in the build directory. Adding another file would mean another path (or overloading the meaning of the plugin path) and just makes the whole thing more fragile and complex.
Having said all that, what would also solve this is either an API for updating LD_LIBRARY_PATH after the program has started; or making setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant which takes a library path as an extra parameter.
Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so? Or does the path have to be chosen dynamically?
To be clear, the situation is: nbdkit (free) -> dlopens nbdkit-vddk-plugin.so (free) -> dlopens libvixDiskLib.so (proprietary) -> dlopens other proprietary plugins -> both libvixDiskLib.so and its plugins have DT_NEEDED "libstdc++.so.X" and other objects that have odd/old compiled versions in its own directory It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK") which has trouble opening its own plugins. I guess you mean adding DT_* to the proprietary library? We don't distribute the proprietary library, end users have to choose to accept the terms of the (very poisonous) license on that library and so they download and install it themselves and just give us a path where they installed it. With all this low quality proprietary software involved I don't expect there's going to be a clean way to solve this, was really only wondering if there's a better way than re-execing the whole program. But we do at least have the re-exec working now.
If you merely want to prevent loading of libstdc++.so or libcrypto.so by vddk, it may be possible to explicitly dlopen DSOs of that name before loading vddk. But there is an existing bug where we do not duplicate properly on soname alone, so we may have to fix that first.
Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
* Richard W. M. Jones:
On Fri, Feb 21, 2020 at 04:00:30PM +0100, Florian Weimer wrote:
* Richard W. M. Jones:
On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
I think what confuses me is that keep talking about a single binary, but clearly there is this separate vddk DSO, and there is talk of plugins. So it seems to me that multiple files are involved already?
nbdkit is a standalone binary that happens to be able to load plugins from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the path for plugins, and there's a wrapper allowing it to get local plugins even when it's still in the build directory. Adding another file would mean another path (or overloading the meaning of the plugin path) and just makes the whole thing more fragile and complex.
Having said all that, what would also solve this is either an API for updating LD_LIBRARY_PATH after the program has started; or making setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant which takes a library path as an extra parameter.
Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so? Or does the path have to be chosen dynamically?
To be clear, the situation is:
nbdkit (free) -> dlopens nbdkit-vddk-plugin.so (free) -> dlopens libvixDiskLib.so (proprietary) -> dlopens other proprietary plugins -> both libvixDiskLib.so and its plugins have DT_NEEDED "libstdc++.so.X" and other objects that have odd/old compiled versions in its own directory
It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK") which has trouble opening its own plugins. I guess you mean adding DT_* to the proprietary library?
No, to nbdkit-vddk-plugin.so. But it looks like have misunderstood the request. Do you want to inhibit loading the libstdc++.so.X from vddk, or the opposite—ensure that it is loaded? The latter obviously taints the process. But maybe that's what you want? Thanks, Florian
On Fri, Feb 21, 2020 at 05:02:12PM +0100, Florian Weimer wrote:
* Richard W. M. Jones:
On Fri, Feb 21, 2020 at 04:00:30PM +0100, Florian Weimer wrote:
* Richard W. M. Jones:
On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
I think what confuses me is that keep talking about a single binary, but clearly there is this separate vddk DSO, and there is talk of plugins. So it seems to me that multiple files are involved already?
nbdkit is a standalone binary that happens to be able to load plugins from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the path for plugins, and there's a wrapper allowing it to get local plugins even when it's still in the build directory. Adding another file would mean another path (or overloading the meaning of the plugin path) and just makes the whole thing more fragile and complex.
Having said all that, what would also solve this is either an API for updating LD_LIBRARY_PATH after the program has started; or making setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant which takes a library path as an extra parameter.
Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so? Or does the path have to be chosen dynamically?
To be clear, the situation is:
nbdkit (free) -> dlopens nbdkit-vddk-plugin.so (free) -> dlopens libvixDiskLib.so (proprietary) -> dlopens other proprietary plugins -> both libvixDiskLib.so and its plugins have DT_NEEDED "libstdc++.so.X" and other objects that have odd/old compiled versions in its own directory
It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK") which has trouble opening its own plugins. I guess you mean adding DT_* to the proprietary library?
No, to nbdkit-vddk-plugin.so.
But it looks like have misunderstood the request.
Do you want to inhibit loading the libstdc++.so.X from vddk, or the opposite—ensure that it is loaded? The latter obviously taints the process. But maybe that's what you want?
Ideally load it but only make it available to VDDK. I think Eric was working on something along these lines, using dlmopen. We're sort of lucky that none of the libraries that VDDK tries to load overlaps with libraries that nbdkit uses (currently). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On 2/21/20 11:42 AM, Richard W.M. Jones wrote:
To be clear, the situation is:
nbdkit (free) -> dlopens nbdkit-vddk-plugin.so (free) -> dlopens libvixDiskLib.so (proprietary) -> dlopens other proprietary plugins -> both libvixDiskLib.so and its plugins have DT_NEEDED "libstdc++.so.X" and other objects that have odd/old compiled versions in its own directory
It's the proprietary library libvixDiskLib.so (colloquially known as "VDDK") which has trouble opening its own plugins. I guess you mean adding DT_* to the proprietary library?
No, to nbdkit-vddk-plugin.so.
But it looks like have misunderstood the request.
Do you want to inhibit loading the libstdc++.so.X from vddk, or the opposite—ensure that it is loaded? The latter obviously taints the process. But maybe that's what you want?
Ideally load it but only make it available to VDDK. I think Eric was working on something along these lines, using dlmopen.
The nbdkit process having its namespace tainted by VDDK code is (so far) a non-issue, as long as the libraries that VDDK ships that override system libraries do not intersect with the set of libraries used by nbdkit itself. (For example, VDDK overrides libstdc++.so, but since nbdkit is written in C rather than C++, we don't care what C++ stuff VDDK drags into the process). But you are right that if, down the road, a future VDDK release poisons even more system libraries, we'd NEED to use dlmopen() to isolate the real system library interfaces used by nbdkit from the out-of-date override library interfaces pulled in by VDDK's DT_NEEDED. However, it was my annoying discovery that dlmopen() is still not a first-class citizen in glibc/gdb that spawned this whole thread in the first place. While I did have a working example using dlmopen() where I was able to hook dlopen() to my heart's content, that hook was not alone sufficient to overcome the DT_NEEDED problems, and it made life a lot tougher since anything in the dlmopen'd space was undebuggable under gdb. Loading the leaf dependencies first before libvixDiskLib.so has no difference on the amount of tainting to the overall nbdkit process compared to letting the leaf dependencies get pulled in via LD_LIBRARY_PATH under re-exec.
We're sort of lucky that none of the libraries that VDDK tries to load overlaps with libraries that nbdkit uses (currently).
Rich.
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On 2/21/20 9:00 AM, Florian Weimer wrote:
* Richard W. M. Jones:
On Fri, Feb 21, 2020 at 01:19:34PM +0100, Florian Weimer wrote:
I think what confuses me is that keep talking about a single binary, but clearly there is this separate vddk DSO, and there is talk of plugins. So it seems to me that multiple files are involved already?
nbdkit is a standalone binary that happens to be able to load plugins from a well-known path, eg nbdkit-vddk-plugin.so. nbdkit knows the path for plugins, and there's a wrapper allowing it to get local plugins even when it's still in the build directory. Adding another file would mean another path (or overloading the meaning of the plugin path) and just makes the whole thing more fragile and complex.
Having said all that, what would also solve this is either an API for updating LD_LIBRARY_PATH after the program has started; or making setenv ("LD_LIBRARY_PATH",...) DTRT*; or some kind of dlopen() variant which takes a library path as an extra parameter.
Have you tried adding DT_RUNPATH or DT_RPATH to nbdkit-vddk-plugin.so?
Post-processing an existing closed-source .so shipped from an external vendor might have negative consequences - while it may be possible to modify the ELF image to add a DT_RUNPATH entry or modify the DT_NEEDED entries to use anchored names based on $ORIGIN rather than bare names, there's no telling if such modification would be in violation of agreements or even cause failure to load if the proprietary code is using shenanigans like validating a checksum of shipped binaries to detect tampering.
Or does the path have to be chosen dynamically?
So, since we cannot fix the existing product, and have no idea if/when the vendor will release an updated version that has saner libraries, a dynamic search path is the only option to getting their product to load (but whether that is done by LD_LIBRARY_PATH, LD_AUDIT, pre-loading libraries, or something else, is where we have a bit more control).
If you merely want to prevent loading of libstdc++.so or libcrypto.so by vddk, it may be possible to explicitly dlopen DSOs of that name before loading vddk. But there is an existing bug where we do not duplicate properly on soname alone, so we may have to fix that first.
The problem then becomes "given an arbitrary libvixDiskLib.so, how do we determine the dependency of bare libraries it would want to load so that we can pre-emptively load those libraries by direct path name first". My re-exec solution was nice - it works for all versions of VDDK. Whereas with scraping the binary to see what DT_NEEDED entries it has is a bit more image-specific: for example, VDDK 5.5.5 loads "libcrypto.so.1.0.0", VDDK-6.5 loads "libcrypto.so.1.0.2". I don't want to hard-code any of those library names into nbdkit-vddk-plugin.so, because that would needlessly tie the open source program into a specific VDDK release, instead of working with all of them. However, if it is easy enough to compute our own topological sort of DT_NEEDED for a given VDDK .so, and then pre-load in leaf-first order, so that by the time we finally get around to dlopen()ing libvixDiskLib.so and calling its init function, then all of the subsequent dlopen()s performed by VDDK code will succeed right off the bat because the library has already been loaded in memory. Or at least, that's what I'm hoping your suggesting. And if we do that, it would avoid the need to re-exec with LD_LIBRARY_PATH set. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
participants (4)
-
Carlos O'Donell -
Eric Blake -
Florian Weimer -
Richard W.M. Jones