[libnbd PATCH v4 0/2] lib/utils: introduce async-signal-safe execvpe()
This is version 4 of the following sub-series: [libnbd PATCH v3 09/29] lib/utils: introduce async-signal-safe execvpe() [libnbd PATCH v3 10/29] lib/utils: add unit tests for async-signal-safe execvpe() http://mid.mail-archive.com/20230215141158.2426855-10-lersek@redhat.com http://mid.mail-archive.com/20230215141158.2426855-11-lersek@redhat.com The Notes section on each patch records the updates for that patch. For assisting with incremental review, here's a range-diff:
1: c5f89eaa0aaf ! 1: 2a3c95d0701f lib/utils: introduce async-signal-safe execvpe() @@ Commit message not to pass it, per APPLICATION USAGE [09], but on Linux/glibc, O_EXEC does not seem supported, only O_PATH does [10].
- Thus the chosen approach -- pre-generate filenames -- contains a small + Thus the chosen approach -- pre-generate filenames -- contains a small TOCTTOU race (highlighted by Eric) after all, but it should be harmless.
Implementation-defined details: @@ Commit message
If PATH is set but empty ("set to null") [02], or PATH is unset and confstr(_CS_PATH) fails or returns no information or returns the empty - string, we fail the initial scanning (!) with ENOENT. This is consistent - with bash's behavior on Linux/glibc: - - $ PATH= ls - bash: ls: No such file or directory + string, we fail the initial scanning (!) with ENOENT.
Details chosen for unspecified behavior:
@@ Commit message
Suggested-by: Eric Blake <eblake@redhat.com> Signed-off-by: Laszlo Ersek <lersek@redhat.com> + Reviewed-by: Eric Blake <eblake@redhat.com> + Reviewed-by: Richard W.M. Jones <rjones@redhat.com> + + + ## Notes ## + v4: + + - remove double space typo from commit message [Eric] + + - remove the allusion to bash compatibility in the + "Implementation-defined details" part of the commit message, where the + latter discusses PATH being "set to null" [Eric] + + - pick up R-b's from Eric and Rich + + - keep the #include "..." list sorted -- #include "checked-overflow.h" + above "minmax.h", not below it + + - in get_path(), replace the FIXME comments with notes that explain why + we don't lock the environment [Eric]
## lib/internal.h ## @@ lib/internal.h: struct command { @@ lib/internal.h: extern void nbd_internal_fork_safe_assert (int result, const cha
## lib/utils.c ## @@ - #include <limits.h> + #include <sys/uio.h>
- #include "minmax.h" + #include "array-size.h" +#include "checked-overflow.h" + #include "minmax.h"
#include "internal.h" - @@ lib/utils.c: nbd_internal_fork_safe_assert (int result, const char *file, long line, - xwrite (STDERR_FILENO, "' failed.\n", 10); + assertion, "' failed.\n", (char *)NULL); abort (); } + @@ lib/utils.c: nbd_internal_fork_safe_assert (int result, const char *file, long l + bool env_path_found; + size_t path_size, path_size2; + -+ /* FIXME: lock the environment. */ ++ /* Note: per POSIX, here we should lock the environment, even just for ++ * getenv(). However, glibc and any other high-quality libc will not be ++ * modifying "environ" during getenv(), and no sane application should modify ++ * the environment after launching threads. ++ */ + path = getenv ("PATH"); + if ((env_path_found = (path != NULL))) + path = strdup (path); -+ /* FIXME: unlock the environment. */ ++ /* This is where we'd unlock the environment. */ + + if (env_path_found) { + /* This handles out-of-memory as well. */ 2: e8fba75ecf93 ! 2: 647a46b965c4 lib/utils: add unit tests for async-signal-safe execvpe() @@ Commit message nbd_internal_fork_safe_execvpe().
Signed-off-by: Laszlo Ersek <lersek@redhat.com> + Reviewed-by: Richard W.M. Jones <rjones@redhat.com> + Reviewed-by: Eric Blake <eblake@redhat.com> + + + ## Notes ## + v4: + + - pick up R-b's from Rich and Eric + + - "errors.c" makes the test case dependent on pthread_getspecific(), so + reflect Eric's commit 742cbd8c7adc ("lib: Use PTHREAD_LIBS where + needed", 2023-03-17), that is, "xxx_LDADD = $(PTHREAD_LIBS)", to this + test case [thanks to Eric for that fixup BTW] + + - replace EXIT trap handler with cleanup_fn [Eric] + + - Create "subdir/f" as a directory, and extend two test scenarios to + show that "subdir/f", even though "f" has search (execute) permission, + results in EACCES (directly), and does not stop advancement through + PATH="...:subdir:..." (indirectly) [Eric]. Use "mkdir + mkdir" for + creating the "f" directory, rather than "mkdir -p", for symmetry with + "mkdir + mkfifo" before, and "mkdir + touch" after. + + - replace "<imperative>, such that <subjunctive>" with "<imperative>, + such that <indicative>" (= s/lead/leads/) [Eric]
## lib/test-fork-safe-execvpe.c (new) ## @@ @@ lib/Makefile.am: pkgconfig_DATA = libnbd.pc
test_fork_safe_assert_SOURCES = \ @@ lib/Makefile.am: test_fork_safe_assert_SOURCES = \ - test-fork-safe-assert.c \ utils.c \ $(NULL) + test_fork_safe_assert_LDADD = $(PTHREAD_LIBS) + +test_fork_safe_execvpe_SOURCES = \ + $(top_srcdir)/common/utils/vector.c \ @@ lib/Makefile.am: test_fork_safe_assert_SOURCES = \ + test-fork-safe-execvpe.c \ + utils.c \ + $(NULL) ++test_fork_safe_execvpe_LDADD = $(PTHREAD_LIBS)
## lib/test-fork-safe-execvpe.sh (new) ## @@ @@ lib/test-fork-safe-execvpe.sh (new) +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + ++. ../tests/functions.sh ++ +set -e + +# Determine the absolute pathname of the execvpe helper binary. The "realpath" @@ lib/test-fork-safe-execvpe.sh (new) + +# Create a temporary directory and change the working directory to it. +tmpd=$(mktemp -d) -+trap 'rm -r -- "$tmpd"' EXIT ++cleanup_fn rm -r -- "$tmpd" +cd "$tmpd" + +# If the "file" parameter of execvpe() is an empty string, then we must fail -- @@ lib/test-fork-safe-execvpe.sh (new) +mkdir fifo +mkfifo fifo/f + ++# Create a directory with a directory in it. ++mkdir subdir ++mkdir subdir/f ++ +# Create a directory with a non-executable file in it. +mkdir nxregf +touch nxregf/f @@ lib/test-fork-safe-execvpe.sh (new) +# the "file" parameter didn't contain a <slash>.) +run "" empty/f; execve_fail empty/f ENOENT +run "" fifo/f; execve_fail fifo/f EACCES ++run "" subdir/f; execve_fail subdir/f EACCES +run "" nxregf/f; execve_fail nxregf/f EACCES +run "" nxregf/f/; execve_fail nxregf/f/ ENOTDIR +run "" symlink/f; execve_fail symlink/f ELOOP @@ lib/test-fork-safe-execvpe.sh (new) +# +# Show that, if the last candidate fails execve() with an error number that +# would not be fatal otherwise, we do get that error number. -+run empty:fifo:nxregf:symlink f -+execve_fail empty/f,fifo/f,nxregf/f,symlink/f ELOOP ++run empty:fifo:subdir:nxregf:symlink f ++execve_fail empty/f,fifo/f,subdir/f,nxregf/f,symlink/f ELOOP + -+# Put a single prefix in PATH, such that it lead to a successful execution. This -+# exercises two things at the same time: (a) that nbd_internal_execvpe_init() -+# produces *one* candidate (i.e., that no <colon> is seen), and (b) that -+# nbd_internal_fork_safe_execvpe() succeeds for the *last* candidate. Repeat the -+# test with "expr" (called "f" under "bin") and the shell script (called "f" -+# under "sh", triggering the ENOEXEC branch). ++# Put a single prefix in PATH, such that it leads to a successful execution. ++# This exercises two things at the same time: (a) that ++# nbd_internal_execvpe_init() produces *one* candidate (i.e., that no <colon> is ++# seen), and (b) that nbd_internal_fork_safe_execvpe() succeeds for the *last* ++# candidate. Repeat the test with "expr" (called "f" under "bin") and the shell ++# script (called "f" under "sh", triggering the ENOEXEC branch). +run bin f 1 + 1; success bin/f,2 +run sh f arg1; success sh/f,"sh/f arg1" +
Thanks for reviewing, Laszlo Laszlo Ersek (2): lib/utils: introduce async-signal-safe execvpe() lib/utils: add unit tests for async-signal-safe execvpe() .gitignore | 1 + lib/Makefile.am | 11 + lib/internal.h | 22 ++ lib/test-fork-safe-execvpe.c | 117 +++++++ lib/test-fork-safe-execvpe.sh | 277 +++++++++++++++ lib/utils.c | 355 ++++++++++++++++++++ 6 files changed, 783 insertions(+) create mode 100644 lib/test-fork-safe-execvpe.c create mode 100755 lib/test-fork-safe-execvpe.sh base-commit: 742cbd8c7adce91eb61b74524df3eb0180799653
execvp() [01] is powerful: - it performs PATH search [02] if necessary, - it falls back to executing the file with the shell as a shell script in case the underlying execv() or execve() fails with ENOEXEC. However, execvp() is not async-signal-safe [03], and so we shouldn't call it in a child process forked from a multi-threaded parent process [04], which the libnbd client application may well be. Introduce three APIs for safely emulating execvp() with execve(): - nbd_internal_execvpe_init(): to be called in the parent process, before fork(). Allocate and format candidate pathnames for execution with execve(), based on PATH. Allocate a buffer for the longer command line that the shell script fallback needs. - nbd_internal_fork_safe_execvpe(): to be called in the child process. Try the candidates formatted previously in sequence with execve(), as long as execve() fails with errors related to pathname resolution / inode-level file type / execution permission. Stop iterating on fatal errors; if the fatal error is ENOEXEC, activate the shell script fallback. As a small added feature, take the environment from an explicit parameter (effectively imitating the GNU-specific execvpe() function -- cf. commit dc64ac5cdd0b, "states: Use execvp instead of execvpe", 2019-11-15). - nbd_internal_execvpe_uninit(): to be called in the parent process, after fork(). Release the resources allocated with nbd_internal_execvpe_init(). The main idea with the above distribution of responsibilities is that any pre-scanning of PATH -- which was suggested by Eric Blake -- should not include activities that are performed by execve() anyway. I've considered and abandoned two approaches: - During scanning (i.e., in nbd_internal_execvpe_init()), check each candidate with access(X_OK) [05]. I dropped this because the informative APPLICATION USAGE section of the same specification [06] advises against it, saying "[a]n application should instead attempt the action itself and handle the [EACCES] error that occurs if the file is not accessible". - During scanning, open each candidate with O_EXEC [07] until one open() succeeds. Save the sole file descriptor for nbd_internal_fork_safe_execvpe(). In the latter, call fexecve() [08], which is free of all error conditions that necessitate iteration over candidates. Still implement the ENOEXEC (shell script) fallback. I dropped this because (a) fexecve() is a quite recent interface (highlighted by Eric); (b) for an application to open a file for execution with fexecve(), it's more robust to pass O_EXEC to open() than not to pass it, per APPLICATION USAGE [09], but on Linux/glibc, O_EXEC does not seem supported, only O_PATH does [10]. Thus the chosen approach -- pre-generate filenames -- contains a small TOCTTOU race (highlighted by Eric) after all, but it should be harmless. Implementation-defined details: - If PATH searching is necessary, but PATH is absent [01], then fall back to confstr(_CS_PATH) [11], similarly to Linux/glibc execvpe() [12]. If PATH is set but empty ("set to null") [02], or PATH is unset and confstr(_CS_PATH) fails or returns no information or returns the empty string, we fail the initial scanning (!) with ENOENT. Details chosen for unspecified behavior: - Hardwire "/bin/sh" as the shell's pathname [01]. [01] https://pubs.opengroup.org/onlinepubs/9699919799/functions/execvp.html [02] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag... [03] https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#ta... [04] https://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html [05] https://pubs.opengroup.org/onlinepubs/9699919799/functions/access.html [06] https://pubs.opengroup.org/onlinepubs/9699919799/functions/access.html#tag_1... [07] https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html [08] https://pubs.opengroup.org/onlinepubs/9699919799/functions/fexecve.html [09] https://pubs.opengroup.org/onlinepubs/9699919799/functions/fexecve.html#tag_... [10] https://man7.org/linux/man-pages/man2/open.2.html [11] https://pubs.opengroup.org/onlinepubs/9699919799/functions/confstr.html [12] https://man7.org/linux/man-pages/man3/execvpe.3.html Suggested-by: Eric Blake <eblake@redhat.com> Signed-off-by: Laszlo Ersek <lersek@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Richard W.M. Jones <rjones@redhat.com> --- Notes: v4: - remove double space typo from commit message [Eric] - remove the allusion to bash compatibility in the "Implementation-defined details" part of the commit message, where the latter discusses PATH being "set to null" [Eric] - pick up R-b's from Eric and Rich - keep the #include "..." list sorted -- #include "checked-overflow.h" above "minmax.h", not below it - in get_path(), replace the FIXME comments with notes that explain why we don't lock the environment [Eric] lib/internal.h | 22 ++ lib/utils.c | 355 ++++++++++++++++++++ 2 files changed, 377 insertions(+) diff --git a/lib/internal.h b/lib/internal.h index 7543f545cd95..35cb5e8994ee 100644 --- a/lib/internal.h +++ b/lib/internal.h @@ -368,6 +368,18 @@ struct command { uint32_t error; /* Local errno value */ }; +struct execvpe { + string_vector pathnames; + + /* Note: "const_string_vector" is not a good type for "sh_argv" below. Even if + * we reserved enough space in a "const_string_vector", + * const_string_vector_append() would still not be async-signal-safe, due to + * the underlying const_string_vector_insert() calling assert(). + */ + char **sh_argv; + size_t num_sh_args; +}; + /* Test if a callback is "null" or not, and set it to null. */ #define CALLBACK_IS_NULL(cb) ((cb).callback == NULL && (cb).free == NULL) #define CALLBACK_IS_NOT_NULL(cb) (! CALLBACK_IS_NULL ((cb))) @@ -542,4 +554,14 @@ extern void nbd_internal_fork_safe_assert (int result, const char *file, __func__, #expression)) #endif +extern int nbd_internal_execvpe_init (struct execvpe *ctx, const char *file, + size_t num_args) + LIBNBD_ATTRIBUTE_NONNULL (1, 2); +extern void nbd_internal_execvpe_uninit (struct execvpe *ctx) + LIBNBD_ATTRIBUTE_NONNULL (1); +extern int nbd_internal_fork_safe_execvpe (struct execvpe *ctx, + const string_vector *argv, + char * const *envp) + LIBNBD_ATTRIBUTE_NONNULL (1, 2, 3); + #endif /* LIBNBD_INTERNAL_H */ diff --git a/lib/utils.c b/lib/utils.c index a8c6c217e2e4..22b466d48258 100644 --- a/lib/utils.c +++ b/lib/utils.c @@ -29,6 +29,7 @@ #include <sys/uio.h> #include "array-size.h" +#include "checked-overflow.h" #include "minmax.h" #include "internal.h" @@ -505,3 +506,357 @@ nbd_internal_fork_safe_assert (int result, const char *file, long line, assertion, "' failed.\n", (char *)NULL); abort (); } + +/* Returns the value of the PATH environment variable -- falling back to + * confstr(_CS_PATH) if PATH is absent -- as a dynamically allocated string. On + * failure, sets "errno" and returns NULL. + */ +static char * +get_path (void) + LIBNBD_ATTRIBUTE_ALLOC_DEALLOC (free) +{ + char *path; + bool env_path_found; + size_t path_size, path_size2; + + /* Note: per POSIX, here we should lock the environment, even just for + * getenv(). However, glibc and any other high-quality libc will not be + * modifying "environ" during getenv(), and no sane application should modify + * the environment after launching threads. + */ + path = getenv ("PATH"); + if ((env_path_found = (path != NULL))) + path = strdup (path); + /* This is where we'd unlock the environment. */ + + if (env_path_found) { + /* This handles out-of-memory as well. */ + return path; + } + + errno = 0; + path_size = confstr (_CS_PATH, NULL, 0); + if (path_size == 0) { + /* If _CS_PATH does not have a configuration-defined value, just store + * ENOENT to "errno". + */ + if (errno == 0) + errno = ENOENT; + + return NULL; + } + + path = malloc (path_size); + if (path == NULL) + return NULL; + + path_size2 = confstr (_CS_PATH, path, path_size); + assert (path_size2 == path_size); + return path; +} + +/* nbd_internal_execvpe_init() and nbd_internal_fork_safe_execvpe() together + * present an execvp() alternative that is async-signal-safe. + * + * nbd_internal_execvpe_init() may only be called before fork(), for filling in + * the caller-allocated, uninitialized "ctx" structure. If + * nbd_internal_execvpe_init() succeeds, then fork() may be called. + * Subsequently, in the child process, nbd_internal_fork_safe_execvpe() may be + * called with the inherited "ctx" structure, while in the parent process, + * nbd_internal_execvpe_uninit() must be called to uninitialize (evacuate) the + * "ctx" structure. + * + * On failure, "ctx" will not have been modified, "errno" is set, and -1 is + * returned. Failures include: + * + * - Errors forwarded from underlying functions such as strdup(), confstr(), + * malloc(), string_vector_append(). + * + * - ENOENT: "file" is an empty string. + * + * - ENOENT: "file" does not contain a <slash> "/" character, the PATH + * environment variable is not set, and confstr() doesn't associate a + * configuration-defined value with _CS_PATH. + * + * - ENOENT: "file" does not contain a <slash> "/" character, and: (a) the PATH + * environment variable is set to the empty string, or (b) PATH is not + * set, and confstr() outputs the empty string for _CS_PATH. + * + * - EOVERFLOW: the sizes or counts of necessary objects could not be expressed. + * + * - EINVAL: "num_args" is less than 2. + * + * On success, the "ctx" structure will have been filled in, and 0 is returned. + * + * - "pathnames" member: + * + * - All strings pointed-to by elements of the "pathnames" string_vector + * member are owned by "pathnames". + * + * - If "file" contains a <slash> "/" character, then the sole entry in + * "pathnames" is a copy of "file". + * + * - If "file" does not contain a <slash> "/" character: + * + * Let "system path" be defined as the value of the PATH environment + * variable, if the latter exists, and as the value output by confstr() for + * _CS_PATH otherwise. Per the ENOENT specifications above, "system path" is + * a non-empty string. Let "system path" further be of the form + * + * <prefix_0> [n = 1] + * + * or + * + * <prefix_0>:<prefix_1>:...:<prefix_(n-1)> [n >= 2] + * + * where for each 0 <= i < n, <prefix_i> does not contain the <colon> ":" + * character. In the (n = 1) case, <prefix_0> is never empty (see ENOENT + * above), while in the (n >= 2) case, any individual <prefix_i> may or may + * not be empty. + * + * The "pathnames" string_vector member has n elements; for each 0 <= i < n, + * element i is of the form + * + * suffix(curdir(<prefix_i>))file + * + * where + * + * curdir(x) := "." if x = "" + * curdir(x) := x otherwise + * + * and + * + * suffix(x) := x if "x" ends with a <slash> "/" + * suffix(x) := x "/" otherwise + * + * This transformation implements the POSIX XBD / PATH environment variable + * semantics, creating candidate pathnames for execution by + * nbd_internal_fork_safe_execvpe(). nbd_internal_fork_safe_execvpe() will + * iterate over the candidate pathnames with execve() until execve() + * succeeds, or fails with an error that is due to neither pathname + * resolution, nor the candidate not being a regular file, nor the candidate + * lacking execution permission. + * + * - The "sh_argv" array member will have at least (num_args + 1) elements + * allocated, and none populated. + * + * (The minimum value of "num_args" is 2 -- see EINVAL above. According to + * POSIX, "[t]he argument /arg0/ should point to a filename string that is + * associated with the process being started by one of the /exec/ functions", + * plus "num_args" includes the null pointer that terminates the argument + * list.) + * + * This allocation is made in anticipation of execve() failing for a + * particular candidate inside nbd_internal_fork_safe_execvpe() with ENOEXEC + * ("[t]he new process image file has the appropriate access permission but + * has an unrecognized format"). While that failure terminates the iteration, + * the failed call + * + * execve (pathnames[i], + * { argv[0], argv[1], ..., NULL }, // (num_args >= 2) elements + * { envp[0], envp[1], ..., NULL }) + * + * must be repeated as + * + * execve (<shell-path>, + * { argv[0], pathnames[i], // ((num_args + 1) >= 3) + argv[1], ..., NULL }, // elements + * { envp[0], envp[1], ..., NULL }) + * + * for emulating execvp(). The allocation in the "sh_argv" member makes it + * possible just to *link* the original "argv" elements and the "pathnames[i]" + * candidate into the right positions. + * + * (POSIX leaves the shell pathname unspecified; "/bin/sh" should be good + * enough.) + * + * The shell *binary* will see itself being executed under the name "argv[0]", + * will receive "pathnames[i]" as the pathname of the shell *script* to read + * and interpret ("command_file" in POSIX terminology), will expose + * "pathnames[i]" as the positional parameter $0 to the script, and will + * forward "argv[1]" and the rest to the script as positional parameters $1 + * and onward. + */ +int +nbd_internal_execvpe_init (struct execvpe *ctx, const char *file, + size_t num_args) +{ + int rc; + char *sys_path; + string_vector pathnames; + char *pathname; + size_t num_sh_args; + char **sh_argv; + size_t sh_argv_bytes; + + rc = -1; + + if (file[0] == '\0') { + errno = ENOENT; + return rc; + } + + /* First phase. */ + sys_path = NULL; + pathnames = (string_vector)empty_vector; + + if (strchr (file, '/') == NULL) { + size_t file_len; + const char *sys_path_element, *scan; + bool finish; + + sys_path = get_path (); + if (sys_path == NULL) + return rc; + + if (sys_path[0] == '\0') { + errno = ENOENT; + goto free_sys_path; + } + + pathname = NULL; + file_len = strlen (file); + sys_path_element = sys_path; + scan = sys_path; + do { + assert (sys_path_element <= scan); + finish = (*scan == '\0'); + if (finish || *scan == ':') { + const char *sys_path_copy_start; + size_t sys_path_copy_size; + size_t sep_copy_size; + size_t pathname_size; + char *p; + + if (scan == sys_path_element) { + sys_path_copy_start = "."; + sys_path_copy_size = 1; + } else { + sys_path_copy_start = sys_path_element; + sys_path_copy_size = scan - sys_path_element; + } + + assert (sys_path_copy_size >= 1); + sep_copy_size = (sys_path_copy_start[sys_path_copy_size - 1] != '/'); + + if (ADD_OVERFLOW (sys_path_copy_size, sep_copy_size, &pathname_size) || + ADD_OVERFLOW (pathname_size, file_len, &pathname_size) || + ADD_OVERFLOW (pathname_size, 1u, &pathname_size)) { + errno = EOVERFLOW; + goto empty_pathnames; + } + + pathname = malloc (pathname_size); + if (pathname == NULL) + goto empty_pathnames; + p = pathname; + + memcpy (p, sys_path_copy_start, sys_path_copy_size); + p += sys_path_copy_size; + + memcpy (p, "/", sep_copy_size); + p += sep_copy_size; + + memcpy (p, file, file_len); + p += file_len; + + *p++ = '\0'; + + if (string_vector_append (&pathnames, pathname) == -1) + goto empty_pathnames; + /* Ownership transferred. */ + pathname = NULL; + + sys_path_element = scan + 1; + } + + ++scan; + } while (!finish); + } else { + pathname = strdup (file); + if (pathname == NULL) + return rc; + + if (string_vector_append (&pathnames, pathname) == -1) + goto empty_pathnames; + /* Ownership transferred. */ + pathname = NULL; + } + + /* Second phase. */ + if (num_args < 2) { + errno = EINVAL; + goto empty_pathnames; + } + if (ADD_OVERFLOW (num_args, 1u, &num_sh_args) || + MUL_OVERFLOW (num_sh_args, sizeof *sh_argv, &sh_argv_bytes)) { + errno = EOVERFLOW; + goto empty_pathnames; + } + sh_argv = malloc (sh_argv_bytes); + if (sh_argv == NULL) + goto empty_pathnames; + + /* Commit. */ + ctx->pathnames = pathnames; + ctx->sh_argv = sh_argv; + ctx->num_sh_args = num_sh_args; + rc = 0; + /* Fall through, for freeing temporaries. */ + +empty_pathnames: + if (rc == -1) { + free (pathname); + string_vector_empty (&pathnames); + } + +free_sys_path: + free (sys_path); + + return rc; +} + +void +nbd_internal_execvpe_uninit (struct execvpe *ctx) +{ + free (ctx->sh_argv); + ctx->num_sh_args = 0; + string_vector_empty (&ctx->pathnames); +} + +int +nbd_internal_fork_safe_execvpe (struct execvpe *ctx, const string_vector *argv, + char * const *envp) +{ + size_t pathname_idx; + + NBD_INTERNAL_FORK_SAFE_ASSERT (ctx->pathnames.len > 0); + + pathname_idx = 0; + do { + (void)execve (ctx->pathnames.ptr[pathname_idx], argv->ptr, envp); + if (errno != EACCES && errno != ELOOP && errno != ENAMETOOLONG && + errno != ENOENT && errno != ENOTDIR) + break; + + ++pathname_idx; + } while (pathname_idx < ctx->pathnames.len); + + if (errno == ENOEXEC) { + char **sh_argp; + size_t argv_idx; + + NBD_INTERNAL_FORK_SAFE_ASSERT (ctx->num_sh_args >= argv->len); + NBD_INTERNAL_FORK_SAFE_ASSERT (ctx->num_sh_args - argv->len == 1); + + sh_argp = ctx->sh_argv; + *sh_argp++ = argv->ptr[0]; + *sh_argp++ = ctx->pathnames.ptr[pathname_idx]; + for (argv_idx = 1; argv_idx < argv->len; ++argv_idx) + *sh_argp++ = argv->ptr[argv_idx]; + + (void)execve ("/bin/sh", ctx->sh_argv, envp); + } + + return -1; +}
Don't try to test async-signal-safety, but strive to exercise as many as possible paths through nbd_internal_execvpe_init() and nbd_internal_fork_safe_execvpe(). Signed-off-by: Laszlo Ersek <lersek@redhat.com> Reviewed-by: Richard W.M. Jones <rjones@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- Notes: v4: - pick up R-b's from Rich and Eric - "errors.c" makes the test case dependent on pthread_getspecific(), so reflect Eric's commit 742cbd8c7adc ("lib: Use PTHREAD_LIBS where needed", 2023-03-17), that is, "xxx_LDADD = $(PTHREAD_LIBS)", to this test case [thanks to Eric for that fixup BTW] - replace EXIT trap handler with cleanup_fn [Eric] - Create "subdir/f" as a directory, and extend two test scenarios to show that "subdir/f", even though "f" has search (execute) permission, results in EACCES (directly), and does not stop advancement through PATH="...:subdir:..." (indirectly) [Eric]. Use "mkdir + mkdir" for creating the "f" directory, rather than "mkdir -p", for symmetry with "mkdir + mkfifo" before, and "mkdir + touch" after. - replace "<imperative>, such that <subjunctive>" with "<imperative>, such that <indicative>" (= s/lead/leads/) [Eric] lib/test-fork-safe-execvpe.c | 117 +++++++++ lib/Makefile.am | 11 + lib/test-fork-safe-execvpe.sh | 277 ++++++++++++++++++++ .gitignore | 1 + 4 files changed, 406 insertions(+) diff --git a/lib/test-fork-safe-execvpe.c b/lib/test-fork-safe-execvpe.c new file mode 100644 index 000000000000..09b91102b613 --- /dev/null +++ b/lib/test-fork-safe-execvpe.c @@ -0,0 +1,117 @@ + /* nbd client library in userspace + * Copyright (C) 2013-2023 Red Hat Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <config.h> + +#include <errno.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +#include "internal.h" + +extern char **environ; + +/* This is a perror() replacement that makes the error message more + * machine-readable, for a select few error numbers. Do not use it as a general + * error() replacement, only upon nbd_internal_execvpe_init() and + * nbd_internal_fork_safe_execvpe() failure. + */ +static void +xperror (const char *s) +{ + const char *err; + + if (s != NULL && *s != '\0') + (void)fprintf (stderr, "%s: ", s); + + switch (errno) { + case EACCES: + err = "EACCES"; + break; + case ELOOP: + err = "ELOOP"; + break; + case ENOENT: + err = "ENOENT"; + break; + case ENOTDIR: + err = "ENOTDIR"; + break; + default: + err = strerror (errno); + } + (void)fprintf (stderr, "%s\n", err); +} + +int +main (int argc, char **argv) +{ + struct execvpe ctx; + const char *prog_file; + string_vector prog_argv; + size_t i; + + if (argc < 3) { + fprintf (stderr, "%1$s: usage: %1$s program-to-exec argv0 ...\n", argv[0]); + return EXIT_FAILURE; + } + + prog_file = argv[1]; + + /* For the argv of the program to execute, we need to drop our argv[0] (= our + * own name) and argv[1] (= the program we need to execute), and to tack on a + * terminating null pointer. Note that "argc" does not include the terminating + * NULL. + */ + prog_argv = (string_vector)empty_vector; + if (string_vector_reserve (&prog_argv, argc - 2 + 1) == -1) { + perror ("string_vector_reserve"); + return EXIT_FAILURE; + } + + for (i = 2; i < argc; ++i) + (void)string_vector_append (&prog_argv, argv[i]); + (void)string_vector_append (&prog_argv, NULL); + + if (nbd_internal_execvpe_init (&ctx, prog_file, prog_argv.len) == -1) { + xperror ("nbd_internal_execvpe_init"); + goto reset_prog_argv; + } + + /* Print out the generated candidates. */ + for (i = 0; i < ctx.pathnames.len; ++i) + (void)fprintf (stdout, "%s\n", ctx.pathnames.ptr[i]); + + if (fflush (stdout) == EOF) { + perror ("fflush"); + goto uninit_execvpe; + } + + (void)nbd_internal_fork_safe_execvpe (&ctx, &prog_argv, environ); + xperror ("nbd_internal_fork_safe_execvpe"); + /* fall through */ + +uninit_execvpe: + nbd_internal_execvpe_uninit (&ctx); + +reset_prog_argv: + string_vector_reset (&prog_argv); + + return EXIT_FAILURE; +} diff --git a/lib/Makefile.am b/lib/Makefile.am index 4fa02feef584..366fb74dbef1 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -33,6 +33,7 @@ EXTRA_DIST = \ $(generator_built) \ libnbd.syms \ test-fork-safe-assert.sh \ + test-fork-safe-execvpe.sh \ $(NULL) lib_LTLIBRARIES = libnbd.la @@ -105,10 +106,12 @@ pkgconfig_DATA = libnbd.pc TESTS = \ test-fork-safe-assert.sh \ + test-fork-safe-execvpe.sh \ $(NULL) check_PROGRAMS = \ test-fork-safe-assert \ + test-fork-safe-execvpe \ $(NULL) test_fork_safe_assert_SOURCES = \ @@ -118,3 +121,11 @@ test_fork_safe_assert_SOURCES = \ utils.c \ $(NULL) test_fork_safe_assert_LDADD = $(PTHREAD_LIBS) + +test_fork_safe_execvpe_SOURCES = \ + $(top_srcdir)/common/utils/vector.c \ + errors.c \ + test-fork-safe-execvpe.c \ + utils.c \ + $(NULL) +test_fork_safe_execvpe_LDADD = $(PTHREAD_LIBS) diff --git a/lib/test-fork-safe-execvpe.sh b/lib/test-fork-safe-execvpe.sh new file mode 100755 index 000000000000..5d2671946c66 --- /dev/null +++ b/lib/test-fork-safe-execvpe.sh @@ -0,0 +1,277 @@ +#!/usr/bin/env bash +# nbd client library in userspace +# Copyright (C) 2013-2023 Red Hat Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +. ../tests/functions.sh + +set -e + +# Determine the absolute pathname of the execvpe helper binary. The "realpath" +# utility is not in POSIX, but Linux, FreeBSD and OpenBSD all have it. +bname=$(basename -- "$0" .sh) +dname=$(dirname -- "$0") +execvpe=$(realpath -- "$dname/$bname") + +# This is an elaborate way to control the PATH variable around the $execvpe +# helper binary as narrowly as possible. +# +# If $1 is "_", then the $execvpe helper binary is invoked with PATH unset. +# Otherwise, the binary is invoked with PATH set to $1. +# +# $2 and onward are passed to $execvpe; note that $2 becomes *both* +# "program-to-exec" for the helper *and* argv[0] for the program executed by the +# helper. +# +# The command itself (including the PATH setting) is written to "cmd" (for error +# reporting purposes only); the standard output and error are saved in "out" and +# "err" respectively; the exit status is written to "status". This function +# should never fail; if it does, then that's a bug in this unit test script, or +# the disk is full etc. +run() +{ + local pathctl=$1 + local program=$2 + local exit_status + + shift 1 + + if test _ = "$pathctl"; then + printf 'unset PATH; %s %s %s\n' "$execvpe" "$program" "$*" >cmd + set +e + ( + unset PATH + "$execvpe" "$program" "$@" >out 2>err + ) + exit_status=$? + set -e + else + printf 'PATH=%s %s %s %s\n' "$pathctl" "$execvpe" "$program" "$*" >cmd + set +e + PATH=$pathctl "$execvpe" "$program" "$@" >out 2>err + exit_status=$? + set -e + fi + printf '%d\n' $exit_status >status +} + +# After "run" returns, the following three functions can verify the result. +# +# Check if the helper binary failed in nbd_internal_execvpe_init(). +# +# $1 is the error number (a macro name such as ENOENT) that's expected of +# nbd_internal_execvpe_init(). +init_fail() +{ + local expected_error="nbd_internal_execvpe_init: $1" + local cmd=$(< cmd) + local err=$(< err) + local status=$(< status) + + if test 0 -eq "$status"; then + printf "'%s' should have failed\\n" "$cmd" >&2 + return 1 + fi + if test x"$err" != x"$expected_error"; then + printf "'%s' should have failed with '%s', got '%s'\\n" \ + "$cmd" "$expected_error" "$err" >&2 + return 1 + fi +} + +# Check if the helper binary failed in nbd_internal_fork_safe_execvpe(). +# +# $1 is the output (the list of candidate pathnames) that +# nbd_internal_execvpe_init() is expected to produce; with inner <newline> +# characters replaced with <comma> characters, and the last <newline> stripped. +# +# $2 is the error number (a macro name such as ENOENT) that's expected of +# nbd_internal_fork_safe_execvpe(). +execve_fail() +{ + local expected_output=$1 + local expected_error="nbd_internal_fork_safe_execvpe: $2" + local cmd=$(< cmd) + local out=$(< out) + local err=$(< err) + local status=$(< status) + + if test 0 -eq "$status"; then + printf "'%s' should have failed\\n" "$cmd" >&2 + return 1 + fi + if test x"$err" != x"$expected_error"; then + printf "'%s' should have failed with '%s', got '%s'\\n" \ + "$cmd" "$expected_error" "$err" >&2 + return 1 + fi + out=${out//$'\n'/,} + if test x"$out" != x"$expected_output"; then + printf "'%s' should have output '%s', got '%s'\\n" \ + "$cmd" "$expected_output" "$out" >&2 + return 1 + fi +} + +# Check if the helper binary and the program executed by it succeeded. +# +# $1 is the output (the list of candidate pathnames) that +# nbd_internal_execvpe_init() is expected to produce, followed by any output +# expected of the program that's executed by the helper; with inner <newline> +# characters replaced with <comma> characters, and the last <newline> stripped. +success() +{ + local expected_output=$1 + local cmd=$(< cmd) + local out=$(< out) + local status=$(< status) + + if test 0 -ne "$status"; then + printf "'%s' should have succeeded\\n" "$cmd" >&2 + return 1 + fi + out=${out//$'\n'/,} + if test x"$out" != x"$expected_output"; then + printf "'%s' should have output '%s', got '%s'\\n" \ + "$cmd" "$expected_output" "$out" >&2 + return 1 + fi +} + +# Create a temporary directory and change the working directory to it. +tmpd=$(mktemp -d) +cleanup_fn rm -r -- "$tmpd" +cd "$tmpd" + +# If the "file" parameter of execvpe() is an empty string, then we must fail -- +# in nbd_internal_execvpe_init() -- regardless of PATH. +run _ ""; init_fail ENOENT +run "" ""; init_fail ENOENT +run . ""; init_fail ENOENT + +# Create subdirectories for triggering non-fatal internal error conditions of +# execvpe(). (Almost) every subdirectory will contain one entry, called "f". +# +# Create a directory that's empty. +mkdir empty + +# Create a directory with a named pipe (FIFO) in it. +mkdir fifo +mkfifo fifo/f + +# Create a directory with a directory in it. +mkdir subdir +mkdir subdir/f + +# Create a directory with a non-executable file in it. +mkdir nxregf +touch nxregf/f + +# Create a symlink loop. +ln -s symlink symlink + +# Create a directory with a (most likely) binary executable in it. +mkdir bin +expr_pathname=$(command -p -v expr) +cp -- "$expr_pathname" bin/f + +# Create a directory with an executable shell script that does not contain a +# shebang (#!). The script will print $0 and $1, and not depend on PATH for it. +mkdir sh +printf "command -p printf '%%s %%s\\\\n' \"\$0\" \"\$1\"\\n" >sh/f +chmod u+x sh/f + +# In the tests below, invoke each "f" such that the "file" parameter of +# execvpe() contain a <slash> character. +# +# Therefore, PATH does not matter. Set it to the empty string. (Which in this +# implementation would cause nbd_internal_execvpe_init() to fail with ENOENT, if +# the "file" parameter didn't contain a <slash>.) +run "" empty/f; execve_fail empty/f ENOENT +run "" fifo/f; execve_fail fifo/f EACCES +run "" subdir/f; execve_fail subdir/f EACCES +run "" nxregf/f; execve_fail nxregf/f EACCES +run "" nxregf/f/; execve_fail nxregf/f/ ENOTDIR +run "" symlink/f; execve_fail symlink/f ELOOP + +# This runs "expr 1 + 1". +run "" bin/f 1 + 1; success bin/f,2 + +# This triggers the ENOEXEC branch in nbd_internal_fork_safe_execvpe(). +# nbd_internal_fork_safe_execvpe() will first try +# +# execve("sh/f", {"sh/f", "arg1", NULL}, envp) +# +# hitting ENOEXEC. Then it will successfully call +# +# execve("/bin/sh", {"sh/f", "sh/f", "arg1", NULL}, envp) +# +# The shell script will get "sh/f" for $0 and "arg1" for $1, and print those +# out. +run "" sh/f arg1; success sh/f,"sh/f arg1" + +# In the tests below, the "file" parameter of execvpe() will not contain a +# <slash> character. +# +# Show that PATH matters that way -- first, trigger an ENOENT in +# nbd_internal_execvpe_init() by setting PATH to the empty string. +run "" expr 1 + 1; init_fail ENOENT + +# Fall back to confstr(_CS_PATH) in nbd_internal_execvpe_init(), by unsetting +# PATH. Verify the generated candidates by invoking "getconf PATH" here, and +# appending "/expr" to each prefix. +expected_output=$( + path=$(command -p getconf PATH) + IFS=: + for p in $path; do + printf '%s/%s\n' $p expr + done + command -p expr 1 + 1 +) +run _ expr 1 + 1; success "${expected_output//$'\n'/,}" + +# Continue with tests where the "file" parameter of execvpe() does not contain a +# <slash> character, but now set PATH to explicit prefix lists. +# +# Show that, if the last candidate fails execve() with an error number that +# would not be fatal otherwise, we do get that error number. +run empty:fifo:subdir:nxregf:symlink f +execve_fail empty/f,fifo/f,subdir/f,nxregf/f,symlink/f ELOOP + +# Put a single prefix in PATH, such that it leads to a successful execution. +# This exercises two things at the same time: (a) that +# nbd_internal_execvpe_init() produces *one* candidate (i.e., that no <colon> is +# seen), and (b) that nbd_internal_fork_safe_execvpe() succeeds for the *last* +# candidate. Repeat the test with "expr" (called "f" under "bin") and the shell +# script (called "f" under "sh", triggering the ENOEXEC branch). +run bin f 1 + 1; success bin/f,2 +run sh f arg1; success sh/f,"sh/f arg1" + +# Demonstrate that the order of candidates matters. The first invocation finds +# "expr" (called "f" under "bin"), the second invocation finds the shell script +# under "sh" (triggering the ENOEXEC branch). +run empty:bin:sh f 1 + 1; success empty/f,bin/f,sh/f,2 +run empty:sh:bin f arg1; success empty/f,sh/f,bin/f,"sh/f arg1" + +# Check the expansion of zero-length prefixes in PATH to ".", plus the +# (non-)insertion of the "/" separator. +run a/: f; execve_fail a/f,./f ENOENT +run :a/ f; execve_fail ./f,a/f ENOENT +run : f; execve_fail ./f,./f ENOENT +pushd bin +run : f 1 + 1; success ./f,./f,2 +popd +run :a/:::b/: f; execve_fail ./f,a/f,./f,./f,b/f,./f ENOENT diff --git a/.gitignore b/.gitignore index 54bc20173f95..9df5d9f9db4c 100644 --- a/.gitignore +++ b/.gitignore @@ -130,6 +130,7 @@ Makefile.in /lib/states.h /lib/test-fork-safe-assert /lib/test-fork-safe-assert.err +/lib/test-fork-safe-execvpe /lib/unlocked.h /libtool /ltmain.sh
On Sun, Mar 19, 2023 at 10:41:37AM +0100, Laszlo Ersek wrote:
This is version 4 of the following sub-series:
[libnbd PATCH v3 09/29] lib/utils: introduce async-signal-safe execvpe() [libnbd PATCH v3 10/29] lib/utils: add unit tests for async-signal-safe execvpe()
http://mid.mail-archive.com/20230215141158.2426855-10-lersek@redhat.com http://mid.mail-archive.com/20230215141158.2426855-11-lersek@redhat.com
The Notes section on each patch records the updates for that patch.
For assisting with incremental review, here's a range-diff:
Thank you; the changes compared to v3 look sane, and my R-b still stands on both patches.
## lib/test-fork-safe-execvpe.sh (new) ## @@ @@ lib/test-fork-safe-execvpe.sh (new) +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + ++. ../tests/functions.sh ++ +set -e + +# Determine the absolute pathname of the execvpe helper binary. The "realpath" @@ lib/test-fork-safe-execvpe.sh (new) + +# Create a temporary directory and change the working directory to it. +tmpd=$(mktemp -d) -+trap 'rm -r -- "$tmpd"' EXIT ++cleanup_fn rm -r -- "$tmpd"
"$tmpd" probably starts with /, such that the -- is not strictly necessary; but keep it (good practice, and saves us in case someone sets $TMPDIR to an unusual relative name and mktemp does not give us a leading / after all). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 3/20/23 20:41, Eric Blake wrote:
On Sun, Mar 19, 2023 at 10:41:37AM +0100, Laszlo Ersek wrote:
This is version 4 of the following sub-series:
[libnbd PATCH v3 09/29] lib/utils: introduce async-signal-safe execvpe() [libnbd PATCH v3 10/29] lib/utils: add unit tests for async-signal-safe execvpe()
http://mid.mail-archive.com/20230215141158.2426855-10-lersek@redhat.com http://mid.mail-archive.com/20230215141158.2426855-11-lersek@redhat.com
The Notes section on each patch records the updates for that patch.
For assisting with incremental review, here's a range-diff:
Thank you; the changes compared to v3 look sane, and my R-b still stands on both patches.
## lib/test-fork-safe-execvpe.sh (new) ## @@ @@ lib/test-fork-safe-execvpe.sh (new) +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + ++. ../tests/functions.sh ++ +set -e + +# Determine the absolute pathname of the execvpe helper binary. The "realpath" @@ lib/test-fork-safe-execvpe.sh (new) + +# Create a temporary directory and change the working directory to it. +tmpd=$(mktemp -d) -+trap 'rm -r -- "$tmpd"' EXIT ++cleanup_fn rm -r -- "$tmpd"
"$tmpd" probably starts with /, such that the -- is not strictly necessary; but keep it (good practice, and saves us in case someone sets $TMPDIR to an unusual relative name and mktemp does not give us a leading / after all).
Series merged as commit range 742cbd8c7adc..0b7172b3cffa. Thank you! Laszlo
On Tue, Mar 21, 2023 at 07:04:59AM +0100, Laszlo Ersek wrote:
On 3/20/23 20:41, Eric Blake wrote:
On Sun, Mar 19, 2023 at 10:41:37AM +0100, Laszlo Ersek wrote:
This is version 4 of the following sub-series:
[libnbd PATCH v3 09/29] lib/utils: introduce async-signal-safe execvpe() [libnbd PATCH v3 10/29] lib/utils: add unit tests for async-signal-safe execvpe()
...
Series merged as commit range 742cbd8c7adc..0b7172b3cffa.
I see you already fixed one pipeline failure due to some gcc versions being more picky about function __attribute__ placement than others. The remaining failures are with alpine Linux, where /bin/expr comes from busybox instead of coreutils, and has the unfortunate quality of having its behavior dependent on argv[0]. Starting from a clean clone, I reproduced it locally with: $ git diff diff --git i/lib/test-fork-safe-execvpe.sh w/lib/test-fork-safe-execvpe.sh index 838bac9..4b3700c 100755 --- i/lib/test-fork-safe-execvpe.sh +++ w/lib/test-fork-safe-execvpe.sh @@ -18,7 +18,7 @@ . ../tests/functions.sh -set -e +set -ex # Determine the absolute pathname of the execvpe helper binary. The "realpath" # utility is not in POSIX, but Linux, FreeBSD and OpenBSD all have it. @@ -155,7 +155,7 @@ success() # Create a temporary directory and change the working directory to it. tmpd=$(mktemp -d) -cleanup_fn rm -r -- "$tmpd" +#cleanup_fn rm -r -- "$tmpd" cd "$tmpd" # If the "file" parameter of execvpe() is an empty string, then we must fail -- $ podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge $ podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo libnbd-alpine-edge bash $ ./configure $ make check $ grep tmpd= lib/test-suite.log + tmpd=/tmp/tmp.EMgKeF $ /tmp/tmp.EMgKeF/bin/f 1 + 1 f: applet not found 0b748c9fe495:~$ So it looks like we need some way to work around busybox' insistance that argv[0] determines which applet to run. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 3/21/23 15:05, Eric Blake wrote:
On Tue, Mar 21, 2023 at 07:04:59AM +0100, Laszlo Ersek wrote:
On 3/20/23 20:41, Eric Blake wrote:
On Sun, Mar 19, 2023 at 10:41:37AM +0100, Laszlo Ersek wrote:
This is version 4 of the following sub-series:
[libnbd PATCH v3 09/29] lib/utils: introduce async-signal-safe execvpe() [libnbd PATCH v3 10/29] lib/utils: add unit tests for async-signal-safe execvpe()
...
Series merged as commit range 742cbd8c7adc..0b7172b3cffa.
I see you already fixed one pipeline failure due to some gcc versions being more picky about function __attribute__ placement than others. The remaining failures are with alpine Linux, where /bin/expr comes from busybox instead of coreutils, and has the unfortunate quality of having its behavior dependent on argv[0].
:) I just wanted to announce on-list that I disabled "lib/test-fork-safe-execvpe.sh" on Alpine Linux because of this :)
Starting from a clean clone, I reproduced it locally with:
$ git diff diff --git i/lib/test-fork-safe-execvpe.sh w/lib/test-fork-safe-execvpe.sh index 838bac9..4b3700c 100755 --- i/lib/test-fork-safe-execvpe.sh +++ w/lib/test-fork-safe-execvpe.sh @@ -18,7 +18,7 @@
. ../tests/functions.sh
-set -e +set -ex
# Determine the absolute pathname of the execvpe helper binary. The "realpath" # utility is not in POSIX, but Linux, FreeBSD and OpenBSD all have it. @@ -155,7 +155,7 @@ success()
# Create a temporary directory and change the working directory to it. tmpd=$(mktemp -d) -cleanup_fn rm -r -- "$tmpd" +#cleanup_fn rm -r -- "$tmpd" cd "$tmpd"
# If the "file" parameter of execvpe() is an empty string, then we must fail --
$ podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge $ podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo libnbd-alpine-edge bash $ ./configure $ make check $ grep tmpd= lib/test-suite.log + tmpd=/tmp/tmp.EMgKeF $ /tmp/tmp.EMgKeF/bin/f 1 + 1 f: applet not found 0b748c9fe495:~$
So it looks like we need some way to work around busybox' insistance that argv[0] determines which applet to run.
I couldn't come up with a reproducer like yours. I couldn't figure out how to *quickly* get an interactive Alpine Linux environment with the test failing, and I also couldn't figure out how to trigger "verbose" test runs on gitlab, without polluting the master branch with debug patches. (I tried forking the project in my own space on gitlab, and pushed a debug patch with just "set -x" onto a non-master branch *there* -- but CI didn't start in response to that.) So, I only installed a new Alpine Linux virtual machine -- what a pain *that* was --, and investigated what /usr/bin/expr was. When I found it was a symlink to /bin/busybox, I started looking for Alpine Linux specific tweaks that could replace busybox (in this role) with a real binary executable "expr" utility. I was relieved to find the following wiki article: https://wiki.alpinelinux.org/wiki/How_to_get_regular_stuff_working which promised -- I thought anyways -- a real coreutils package. Imagine my dismay when I found that, after installing coreutils with apk in the Alpine Linux VM, the symlink stayed in place, only its target binary changed from "/bin/busybox" to "coreutils". Well done, Alpine Linux, well done. So, no, this mess (= Alpine Linux) is not salvageable. The "lib/test-fork-safe-execvpe.sh" script depends on "expr" being functional under the name "f". And yes I want "f" to be a single-character filename; otherwise the nice tabular formatting of the script falls apart (or produces overlong lines). As last step, I learned about ci/skipped_tests, and used it to disable the test on alpine linux. The latest pipeline passed: <https://gitlab.com/nbdkit/libnbd/-/pipelines/813280321>. Either way, I wanted to highlight the following commits on the list: 1 b29ff42e5d00 lib: account for realpath deficiency on some platforms 2 65631e5dfff5 lib/utils: try to placate attribute placement restriction from gcc 3 4cae20ccefaf Revert "lib: account for realpath deficiency on some platforms" 4 f5a065aa3a9c ci: skip "lib/test-fork-safe-execvpe.sh" on Alpine Linux Thanks for investigating! Laszlo
On Tue, Mar 21, 2023 at 03:56:22PM +0100, Laszlo Ersek wrote:
$ podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge $ podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo libnbd-alpine-edge bash $ ./configure $ make check $ grep tmpd= lib/test-suite.log + tmpd=/tmp/tmp.EMgKeF $ /tmp/tmp.EMgKeF/bin/f 1 + 1 f: applet not found 0b748c9fe495:~$
So it looks like we need some way to work around busybox' insistance that argv[0] determines which applet to run.
I couldn't come up with a reproducer like yours. I couldn't figure out how to *quickly* get an interactive Alpine Linux environment with the test failing, and I also couldn't figure out how to trigger "verbose" test runs on gitlab, without polluting the master branch with debug patches. (I tried forking the project in my own space on gitlab, and pushed a debug patch with just "set -x" onto a non-master branch *there* -- but CI didn't start in response to that.)
ci/README.rst is informative on that regards; I'm impressed at what lcitool (from the libvirt-ci) project gives us for local reproducibility of repeating various CI build failures.
So, I only installed a new Alpine Linux virtual machine -- what a pain *that* was --, and investigated what /usr/bin/expr was. When I found it was a symlink to /bin/busybox, I started looking for Alpine Linux specific tweaks that could replace busybox (in this role) with a real binary executable "expr" utility.
I was relieved to find the following wiki article:
https://wiki.alpinelinux.org/wiki/How_to_get_regular_stuff_working
which promised -- I thought anyways -- a real coreutils package.
Imagine my dismay when I found that, after installing coreutils with apk in the Alpine Linux VM, the symlink stayed in place, only its target binary changed from "/bin/busybox" to "coreutils". Well done, Alpine Linux, well done.
Yeah, I remember the debates back when GNU coreutils was offered patches to choose to build a multiname binary - it fits with Alpine's goals and matches busybox's minimalist footprint, but it violates GNU Coding Standards (which state that in general, an application's behavior should not depend on argv[0], precisely so that renaming it doesn't change its behavior).
So, no, this mess (= Alpine Linux) is not salvageable. The "lib/test-fork-safe-execvpe.sh" script depends on "expr" being functional under the name "f". And yes I want "f" to be a single-character filename; otherwise the nice tabular formatting of the script falls apart (or produces overlong lines).
As last step, I learned about ci/skipped_tests, and used it to disable the test on alpine linux.
That works, but is not always my favorite: it fixes CI, but does not help any developer on a similar platform. I probably would have tried this in the test itself (untested here): cp /bin/expr f cleanup_fn rm f requires ./f 1 + 1 # We insist on expr behavior independent of argv[0] which will then skip the test (exit status 77) if a renamed /bin/expr doesn't behave the same as the original. But since we haven't heard complaints from anyone using Alpine as their primary development box (or any other non-Alpine but otherwise-busybox setup), we can make that change when such a complaint arises, rather than adding even more churn to your latest series of patches. Another thought - instead of skipping the test when /bin/expr is underwhelming when renamed, we could rewrite the test to avoid that dependency altogether. After all, we are building a test-fork-safe-execvpe binary that does exactly what we say it should; we could hack up test-fork-safe-execvpe.c to depend on argv[0] (I know - using the very thing I don't like about busybox) so that when run by its usual name it performs the unit test, but when run by the basename of f, it does just enough for us to prove that we indeed executed our desired binary (whether that be emulating a subset of /bin/expr, or something else). But again, that's more effort to code up than just skipping the test on a platform that's less likely to be a primary development station in the first place.
The latest pipeline passed: <https://gitlab.com/nbdkit/libnbd/-/pipelines/813280321>.
Either way, I wanted to highlight the following commits on the list:
1 b29ff42e5d00 lib: account for realpath deficiency on some platforms
I didn't even pay attention to this one before you pointed it out; but it is indeed a bug in busybox now that POSIX is moving towards standardizing realpath, so I've filed it: https://bugs.busybox.net/show_bug.cgi?id=15466
2 65631e5dfff5 lib/utils: try to placate attribute placement restriction from gcc 3 4cae20ccefaf Revert "lib: account for realpath deficiency on some platforms" 4 f5a065aa3a9c ci: skip "lib/test-fork-safe-execvpe.sh" on Alpine Linux
Thanks for investigating! Laszlo
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Hi Eric, lots of great stuff in your email, thanks! Let me respond: On 3/21/23 18:28, Eric Blake wrote:
On Tue, Mar 21, 2023 at 03:56:22PM +0100, Laszlo Ersek wrote:
$ podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge $ podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo libnbd-alpine-edge bash $ ./configure $ make check $ grep tmpd= lib/test-suite.log + tmpd=/tmp/tmp.EMgKeF $ /tmp/tmp.EMgKeF/bin/f 1 + 1 f: applet not found 0b748c9fe495:~$
So it looks like we need some way to work around busybox' insistance that argv[0] determines which applet to run.
I couldn't come up with a reproducer like yours. I couldn't figure out how to *quickly* get an interactive Alpine Linux environment with the test failing, and I also couldn't figure out how to trigger "verbose" test runs on gitlab, without polluting the master branch with debug patches. (I tried forking the project in my own space on gitlab, and pushed a debug patch with just "set -x" onto a non-master branch *there* -- but CI didn't start in response to that.)
ci/README.rst is informative on that regards; I'm impressed at what lcitool (from the libvirt-ci) project gives us for local reproducibility of repeating various CI build failures.
Indeed very informative. I was hoping I could skimp on reading it, but now I've bitten the bullet. It does offer precisely such podman commands (two of them) that I need for seamless testing in an Alpine Linux container. Unfortunately, podman does not work for me on RHEL-9.1 as a mere user! podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge it prints: Error: error creating tmpdir: mkdir /run/user/500: permission denied I've searched the web -- I've found nothing conclusive (for example, I don't have a .config/containers/storage.conf file, so its contents cannot be blamed for the misbehavior). However, at least I've determined that Giuseppe Scrivano is an expert in this area, so I've emailed him now. We'll see where that goes.
So, I only installed a new Alpine Linux virtual machine -- what a pain *that* was --, and investigated what /usr/bin/expr was. When I found it was a symlink to /bin/busybox, I started looking for Alpine Linux specific tweaks that could replace busybox (in this role) with a real binary executable "expr" utility.
I was relieved to find the following wiki article:
https://wiki.alpinelinux.org/wiki/How_to_get_regular_stuff_working
which promised -- I thought anyways -- a real coreutils package.
Imagine my dismay when I found that, after installing coreutils with apk in the Alpine Linux VM, the symlink stayed in place, only its target binary changed from "/bin/busybox" to "coreutils". Well done, Alpine Linux, well done.
Yeah, I remember the debates back when GNU coreutils was offered patches to choose to build a multiname binary - it fits with Alpine's goals and matches busybox's minimalist footprint, but it violates GNU Coding Standards (which state that in general, an application's behavior should not depend on argv[0], precisely so that renaming it doesn't change its behavior).
Haha, I didn't expect this. Background that's good to know!
So, no, this mess (= Alpine Linux) is not salvageable. The "lib/test-fork-safe-execvpe.sh" script depends on "expr" being functional under the name "f". And yes I want "f" to be a single-character filename; otherwise the nice tabular formatting of the script falls apart (or produces overlong lines).
As last step, I learned about ci/skipped_tests, and used it to disable the test on alpine linux.
That works, but is not always my favorite: it fixes CI, but does not help any developer on a similar platform. I probably would have tried this in the test itself (untested here):
cp /bin/expr f cleanup_fn rm f requires ./f 1 + 1 # We insist on expr behavior independent of argv[0]
which will then skip the test (exit status 77) if a renamed /bin/expr doesn't behave the same as the original. But since we haven't heard complaints from anyone using Alpine as their primary development box (or any other non-Alpine but otherwise-busybox setup), we can make that change when such a complaint arises, rather than adding even more churn to your latest series of patches.
Another thought - instead of skipping the test when /bin/expr is underwhelming when renamed, we could rewrite the test to avoid that dependency altogether. After all, we are building a test-fork-safe-execvpe binary that does exactly what we say it should; we could hack up test-fork-safe-execvpe.c to depend on argv[0] (I know - using the very thing I don't like about busybox) so that when run by its usual name it performs the unit test, but when run by the basename of f, it does just enough for us to prove that we indeed executed our desired binary (whether that be emulating a subset of /bin/expr, or something else). But again, that's more effort to code up than just skipping the test on a platform that's less likely to be a primary development station in the first place.
This morning I have thought of two other approaches: 1. Simply write another test program in C that prints "hello world" to the standard output. Totally deterministic behavior, a binary executable, and does not depend on any system-provided utility. (The original reason for using "expr" was to avoid the need for such a C source file.) 2. I've recalled that "test-fork-safe-execvpe.c" actually takes argv[0] for the program to execute, as another explicit argument. Correspondingly, the "run" function in "test-fork-safe-execvpe.sh" passes its $2 positional parameter *twice* to the "test-fork-safe-execvpe" binary: # $2 and onward are passed to $execvpe; note that $2 becomes *both* # "program-to-exec" for the helper *and* argv[0] for the program executed by the # helper. I *might* be able to add a "run0" function that does not perform this duplication, i.e., it lets the caller pass in argv[0] separately from "program-to-exec". Then (by my count) I should have to modify only 4 invocations of the form "f 1 + 1"; and in advance I hope that such a change wouldn't destroy the formatting of the script.
The latest pipeline passed: <https://gitlab.com/nbdkit/libnbd/-/pipelines/813280321>.
Either way, I wanted to highlight the following commits on the list:
1 b29ff42e5d00 lib: account for realpath deficiency on some platforms
I didn't even pay attention to this one before you pointed it out; but it is indeed a bug in busybox now that POSIX is moving towards standardizing realpath, so I've filed it: https://bugs.busybox.net/show_bug.cgi?id=15466
Thanks! I remember Austin Group bug #1457 from your earlier messages. Back then I noticed that said ticket called for a utility named "readlink", not "realpath". So it's not exactly that "realpath" (which I believe will remain non-standard?) should accept "--", but that a "readlink" utility, accepting "--", should exist. ... Ah wait, no, that's not right. Upon reading the ticket more carefully now, there is a late comment in the ticket that *does* mandate "realpath": <https://www.austingroupbugs.net/view.php?id=1457#c5898>. OK then! :) (The "Summary" field on Austin Group bug #1457 might be worth updating!) Thanks! Laszlo
2 65631e5dfff5 lib/utils: try to placate attribute placement restriction from gcc 3 4cae20ccefaf Revert "lib: account for realpath deficiency on some platforms" 4 f5a065aa3a9c ci: skip "lib/test-fork-safe-execvpe.sh" on Alpine Linux
Thanks for investigating! Laszlo
On 3/21/23 18:28, Eric Blake wrote:
it is indeed a bug in busybox now that POSIX is moving towards standardizing realpath, so I've filed it: https://bugs.busybox.net/show_bug.cgi?id=15466
I've found another busybox bug. The "/bin/sh" utility is provided by busybox as well (via the usual symlinking). Per POSIX, if execvp(file, { argv[0], argv[1], ..., NULL }) were to fail with -1/ENOEXEC, then execvp() must retry "as if" with execv(<shell path>, { argv[0], file, argv[1], ..., NULL }) In other words, if direct execution of "file" failed because "file" "has the appropriate access permission but has an unrecognized format", then execvp() is required to try executing "file" as a shell script. For that, <shell path> is left unspecified by POSIX, but the arguments of the shell are specified: - Argv[0] remains the same. That is, what we wanted "file" to know itself as, is what we now want *the shell executable* to know itself as. - argv[1] becomes "file" -- this is the script that the shell is supposed to run. - argv[2] and onwards become positional parameters $1, $2, ... for the shell script. And the argv[0] specification is what's violated by busybox, because if argv[0] is anything other than "sh", then the busybox binary doesn't recognize itself as the shell! The simplest way to demonstrate the bug is this: bash-5.2$ ( exec -a foobar /bin/sh <<< "echo hello" ) foobar: applet not found And then, another way to demonstrate the same busybox issue... lets us, in fact, discover a musl bug in turn!!! Consider the following C program (called "test-execvp.c"; the binary is called "test-execvp"): ------------- #include <stdio.h> #include <unistd.h> int main(void) { char *args[] = { "foobar", NULL }; execvp("hello.sh", args); perror("execvp"); return 1; } ------------- The file "hello.sh" resides in the current directory (same directory where "test-execvp" resides). Furthermore it has execute permission, and the following contents: ------- echo hello ------- Now consider the following command (from bash): $ PATH=.:$PATH test-execvp What is supposed to happen is this: (1) bash shall find test-execvp in the current directory per PATH, (2) execvp() shall find "hello.sh" in the current directory per PATH, (3) execvp() shall hit an internal failure -1/ENOEXEC, (4) execvp() shall then invoke the shell (under an unspecified pathname), (5) the shell shall get "foobar" for its argv[0], and "hello.sh" for its argv[1] (6) we shall see "hello" on the standard output. That's exactly what happens on Linux/glibc. (Note: this result has absolutely nothing to do with my execvpe() implementation, or libnbd in the first place.) Now, according to my above description of the busybox bug, we're tempted to believe that step (6) fails on Alpine Linux (using musl + busybox). We expect the busybox binary to be launched, via the /bin/sh symlink, in step (4), and we expect it to fail after step (5), due to it not recognizing "foobar" as an "applet name". It turns out however that step (4) does not happen. musl does not handle ENOEXEC:
bash-5.2$ PATH=.:$PATH test-execvp execvp: Exec format error
execvp() is not permitted by POSIX to fail with ENOEXEC. (Not considering the virtually impossible situation when executing the system shell binary *itself* fails with ENOEXEC.) I've checked the musl source too, at commit 7d756e1c04de ("dns: prefer monotonic clock for timeouts", 2023-02-12). The execvp() implementation: https://git.musl-libc.org/cgit/musl/tree/src/process/execvp.c does not handle ENOEXEC; what's more, the entire tree only sets errno=ENOEXEC in "ldso/dynlink.c". Laszlo
On 3/22/23 15:45, Laszlo Ersek wrote:
On 3/21/23 18:28, Eric Blake wrote:
it is indeed a bug in busybox now that POSIX is moving towards standardizing realpath, so I've filed it: https://bugs.busybox.net/show_bug.cgi?id=15466
I've found another busybox bug.
The "/bin/sh" utility is provided by busybox as well (via the usual symlinking).
Per POSIX, if
execvp(file, { argv[0], argv[1], ..., NULL })
were to fail with -1/ENOEXEC, then execvp() must retry "as if" with
execv(<shell path>, { argv[0], file, argv[1], ..., NULL })
In other words, if direct execution of "file" failed because "file" "has the appropriate access permission but has an unrecognized format", then execvp() is required to try executing "file" as a shell script. For that, <shell path> is left unspecified by POSIX, but the arguments of the shell are specified:
- Argv[0] remains the same. That is, what we wanted "file" to know itself as, is what we now want *the shell executable* to know itself as.
- argv[1] becomes "file" -- this is the script that the shell is supposed to run.
- argv[2] and onwards become positional parameters $1, $2, ... for the shell script.
And the argv[0] specification is what's violated by busybox, because if argv[0] is anything other than "sh", then the busybox binary doesn't recognize itself as the shell!
The simplest way to demonstrate the bug is this:
bash-5.2$ ( exec -a foobar /bin/sh <<< "echo hello" ) foobar: applet not found
And then, another way to demonstrate the same busybox issue... lets us, in fact, discover a musl bug in turn!!!
Consider the following C program (called "test-execvp.c"; the binary is called "test-execvp"):
------------- #include <stdio.h> #include <unistd.h>
int main(void) { char *args[] = { "foobar", NULL };
execvp("hello.sh", args); perror("execvp"); return 1; } -------------
The file "hello.sh" resides in the current directory (same directory where "test-execvp" resides). Furthermore it has execute permission, and the following contents:
------- echo hello -------
Now consider the following command (from bash):
$ PATH=.:$PATH test-execvp
What is supposed to happen is this:
(1) bash shall find test-execvp in the current directory per PATH, (2) execvp() shall find "hello.sh" in the current directory per PATH, (3) execvp() shall hit an internal failure -1/ENOEXEC, (4) execvp() shall then invoke the shell (under an unspecified pathname), (5) the shell shall get "foobar" for its argv[0], and "hello.sh" for its argv[1] (6) we shall see "hello" on the standard output.
That's exactly what happens on Linux/glibc. (Note: this result has absolutely nothing to do with my execvpe() implementation, or libnbd in the first place.)
Now, according to my above description of the busybox bug, we're tempted to believe that step (6) fails on Alpine Linux (using musl + busybox). We expect the busybox binary to be launched, via the /bin/sh symlink, in step (4), and we expect it to fail after step (5), due to it not recognizing "foobar" as an "applet name".
It turns out however that step (4) does not happen. musl does not handle ENOEXEC:
bash-5.2$ PATH=.:$PATH test-execvp execvp: Exec format error
execvp() is not permitted by POSIX to fail with ENOEXEC. (Not considering the virtually impossible situation when executing the system shell binary *itself* fails with ENOEXEC.)
I've checked the musl source too, at commit 7d756e1c04de ("dns: prefer monotonic clock for timeouts", 2023-02-12). The execvp() implementation:
https://git.musl-libc.org/cgit/musl/tree/src/process/execvp.c
does not handle ENOEXEC; what's more, the entire tree only sets errno=ENOEXEC in "ldso/dynlink.c".
I considered reporting a musl bug, but: - per <https://wiki.musl-libc.org/reporting-bugs.html>, musl issues should be reported on the mailing list, - on the mailing list, I've found two reports of this symptom, one from 2018, another from 2020: https://www.openwall.com/lists/musl/2018/03/09/2 https://www.openwall.com/lists/musl/2020/02/12/9 So it turns out that my execvp[e]() implementation is not only async-signal-safe, but mitigates a known bug in musl, too. (BTW what Rich Felker seems to be missing, under the second link above, is that execvp() is *NOT* required to be async-signal-safe (approx. "safe to use from a vforked child"), so malloc() is fair game within execvp(), even in case malloc() is also async-signal-UNSAFE in musl.) Laszlo
On 3/22/23 15:45, Laszlo Ersek wrote:
On 3/21/23 18:28, Eric Blake wrote:
it is indeed a bug in busybox now that POSIX is moving towards standardizing realpath, so I've filed it: https://bugs.busybox.net/show_bug.cgi?id=15466
I've found another busybox bug.
The "/bin/sh" utility is provided by busybox as well (via the usual symlinking).
Per POSIX, if
execvp(file, { argv[0], argv[1], ..., NULL })
were to fail with -1/ENOEXEC, then execvp() must retry "as if" with
execv(<shell path>, { argv[0], file, argv[1], ..., NULL })
In other words, if direct execution of "file" failed because "file" "has the appropriate access permission but has an unrecognized format", then execvp() is required to try executing "file" as a shell script. For that, <shell path> is left unspecified by POSIX, but the arguments of the shell are specified:
- Argv[0] remains the same. That is, what we wanted "file" to know itself as, is what we now want *the shell executable* to know itself as.
- argv[1] becomes "file" -- this is the script that the shell is supposed to run.
- argv[2] and onwards become positional parameters $1, $2, ... for the shell script.
And the argv[0] specification is what's violated by busybox, because if argv[0] is anything other than "sh", then the busybox binary doesn't recognize itself as the shell!
The simplest way to demonstrate the bug is this:
bash-5.2$ ( exec -a foobar /bin/sh <<< "echo hello" ) foobar: applet not found
And then, another way to demonstrate the same busybox issue... lets us, in fact, discover a musl bug in turn!!!
Consider the following C program (called "test-execvp.c"; the binary is called "test-execvp"):
------------- #include <stdio.h> #include <unistd.h>
int main(void) { char *args[] = { "foobar", NULL };
execvp("hello.sh", args); perror("execvp"); return 1; } -------------
The file "hello.sh" resides in the current directory (same directory where "test-execvp" resides). Furthermore it has execute permission, and the following contents:
------- echo hello -------
Now consider the following command (from bash):
$ PATH=.:$PATH test-execvp
What is supposed to happen is this:
(1) bash shall find test-execvp in the current directory per PATH, (2) execvp() shall find "hello.sh" in the current directory per PATH, (3) execvp() shall hit an internal failure -1/ENOEXEC, (4) execvp() shall then invoke the shell (under an unspecified pathname), (5) the shell shall get "foobar" for its argv[0], and "hello.sh" for its argv[1] (6) we shall see "hello" on the standard output.
That's exactly what happens on Linux/glibc. (Note: this result has absolutely nothing to do with my execvpe() implementation, or libnbd in the first place.)
Now, according to my above description of the busybox bug, we're tempted to believe that step (6) fails on Alpine Linux (using musl + busybox). We expect the busybox binary to be launched, via the /bin/sh symlink, in step (4), and we expect it to fail after step (5), due to it not recognizing "foobar" as an "applet name".
It turns out however that step (4) does not happen. musl does not handle ENOEXEC:
It's getting crazier by the hour. I thought to create a reproducer for busybox, in spite of musl breaking down at an earlier point (see above). For that, I *statically linked* "test-execvp" on RHEL-9.1 (using glibc), and then executed the binary in the Alpine Linux container. This should eliminate musl from the picture, and exercise the ENOEXEC fallback (from glibc), invoking /bin/sh (aka busybox) under argv[0] "foobar", and trigger the "unknown applet" bug in busybox. However, this does not happen. Instead, I get "hello" printed. How is that possible? The solution is that glibc *too* has a bug, and that bug hides the busybox bug. Namely, in glibc, going back to historical commit commit 6a032d81581978187f562e5533a32e0a6a3d352b (tag: cvs/libc-960210) Author: Roland McGrath <roland@gnu.org> Date: Sat Feb 10 10:00:27 1996 +0000 Sat Feb 10 04:18:48 1996 Roland McGrath <roland@churchy.gnu.ai.mit.edu> * posix/execvp.c: If execv fails with ENOEXEC, run the shell on the file. Fri Feb 9 11:46:45 1996 Roland McGrath <roland@churchy.gnu.ai.mit.edu> * time/Makefile (CFLAGS-zdump.c, CFLAGS-zic.c, CFLAGS-ialloc.c, CFLAGS-scheck.c): Use -DNOID instead of -Wno-unused. * hurd/Makefile (user-interfaces): Added hurd/tioctl. (note the date: 1996!), the POSIX-mandated fallback execv(<shell path>, { argv[0], file, argv[1], ..., NULL }) is not being done. Instead, the following is done: execv(<shell path>, { <shell path>, file, argv[1], ..., NULL }) In other words, the original argv[0] is not preserved, but is replaced by <shell path>. (Look for _PATH_BSHELL in said historical glibc commit, and also in today's glibc file "posix/execvpe.c".) This can be demonstrated with: $ PATH=.:$PATH strace -etrace=execve test-execvp execve("./test-execvp", ["test-execvp"], 0x7ffc0d1e5248 /* 85 vars */) = 0 execve("./hello.sh", ["foobar"], 0x7ffc528e14a8 /* 85 vars */) = -1 ENOEXEC (Exec format error) execve("/bin/sh", ["/bin/sh", "./hello.sh"], 0x7ffc528e14a8 /* 85 vars */) = 0 hello +++ exited with 0 +++ The third execve() call should be: execve("/bin/sh", ["foobar", "./hello.sh"], 0x7ffc528e14a8 /* 85 vars */) = 0 ^^^^^^^^ Laszlo
On Wed, Mar 22, 2023 at 04:53:42PM +0100, Laszlo Ersek wrote: ...
What is supposed to happen is this:
(1) bash shall find test-execvp in the current directory per PATH, (2) execvp() shall find "hello.sh" in the current directory per PATH, (3) execvp() shall hit an internal failure -1/ENOEXEC, (4) execvp() shall then invoke the shell (under an unspecified pathname), (5) the shell shall get "foobar" for its argv[0], and "hello.sh" for its argv[1] (6) we shall see "hello" on the standard output.
That's exactly what happens on Linux/glibc. (Note: this result has absolutely nothing to do with my execvpe() implementation, or libnbd in the first place.)
Now, according to my above description of the busybox bug, we're tempted to believe that step (6) fails on Alpine Linux (using musl + busybox). We expect the busybox binary to be launched, via the /bin/sh symlink, in step (4), and we expect it to fail after step (5), due to it not recognizing "foobar" as an "applet name".
It turns out however that step (4) does not happen. musl does not handle ENOEXEC:
It's getting crazier by the hour.
No joke. Locally, I wrote: $ cat foo echo "hello from $0:$@" $ chmod +x foo $ bash ./foo foobar arg1 hello from ./foo:foobar arg1 $ bash -c ./foo foobar arg1 hello from ./foo: $ bash -c 'exec -a "$0" ./foo "$@"' foobar arg1 hello from foobar:arg1 $ bash -c '. ./foo' foobar arg1 hello from foobar:arg1 The latter two match the behavior we want to see. Alas, POSIX has not yet standardized 'exec -a', but the ability to use 'sh -c' to set argv0, plus . to source a file as shell commands, gets us to the same place.
I thought to create a reproducer for busybox, in spite of musl breaking down at an earlier point (see above). For that, I *statically linked* "test-execvp" on RHEL-9.1 (using glibc), and then executed the binary in the Alpine Linux container. This should eliminate musl from the picture, and exercise the ENOEXEC fallback (from glibc), invoking /bin/sh (aka busybox) under argv[0] "foobar", and trigger the "unknown applet" bug in busybox.
However, this does not happen. Instead, I get "hello" printed. How is that possible?
The solution is that glibc *too* has a bug, and that bug hides the busybox bug. Namely, in glibc, going back to historical commit
commit 6a032d81581978187f562e5533a32e0a6a3d352b (tag: cvs/libc-960210) Author: Roland McGrath <roland@gnu.org> Date: Sat Feb 10 10:00:27 1996 +0000
Sat Feb 10 04:18:48 1996 Roland McGrath <roland@churchy.gnu.ai.mit.edu>
* posix/execvp.c: If execv fails with ENOEXEC, run the shell on the file.
Fri Feb 9 11:46:45 1996 Roland McGrath <roland@churchy.gnu.ai.mit.edu>
* time/Makefile (CFLAGS-zdump.c, CFLAGS-zic.c, CFLAGS-ialloc.c, CFLAGS-scheck.c): Use -DNOID instead of -Wno-unused.
* hurd/Makefile (user-interfaces): Added hurd/tioctl.
(note the date: 1996!), the POSIX-mandated fallback
execv(<shell path>, { argv[0], file, argv[1], ..., NULL })
is not being done. Instead, the following is done:
execv(<shell path>, { <shell path>, file, argv[1], ..., NULL })
In other words, the original argv[0] is not preserved, but is replaced by <shell path>. (Look for _PATH_BSHELL in said historical glibc commit, and also in today's glibc file "posix/execvpe.c".)
Wow. This is so historically old that it's probably worth asking on the Austin Group if argv[0] must be preserved or can be changed to the shell name, and/or whether the standard should add support for 'exec -a argv0 cmd args' as a way to set argv[0] despite this long-standing glibc behavior. I'll reply back when I have some URLs to list archives of my pending questions.
This can be demonstrated with:
$ PATH=.:$PATH strace -etrace=execve test-execvp
execve("./test-execvp", ["test-execvp"], 0x7ffc0d1e5248 /* 85 vars */) = 0 execve("./hello.sh", ["foobar"], 0x7ffc528e14a8 /* 85 vars */) = -1 ENOEXEC (Exec format error) execve("/bin/sh", ["/bin/sh", "./hello.sh"], 0x7ffc528e14a8 /* 85 vars */) = 0 hello +++ exited with 0 +++
The third execve() call should be:
execve("/bin/sh", ["foobar", "./hello.sh"], 0x7ffc528e14a8 /* 85 vars */) = 0 ^^^^^^^^
Laszlo
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Wed, Mar 22, 2023 at 04:53:42PM +0100, Laszlo Ersek wrote:
The solution is that glibc *too* has a bug, and that bug hides the busybox bug. Namely, in glibc, going back to historical commit
commit 6a032d81581978187f562e5533a32e0a6a3d352b (tag: cvs/libc-960210) Author: Roland McGrath <roland@gnu.org> Date: Sat Feb 10 10:00:27 1996 +0000
Sat Feb 10 04:18:48 1996 Roland McGrath <roland@churchy.gnu.ai.mit.edu>
* posix/execvp.c: If execv fails with ENOEXEC, run the shell on the file.
Fri Feb 9 11:46:45 1996 Roland McGrath <roland@churchy.gnu.ai.mit.edu>
* time/Makefile (CFLAGS-zdump.c, CFLAGS-zic.c, CFLAGS-ialloc.c, CFLAGS-scheck.c): Use -DNOID instead of -Wno-unused.
* hurd/Makefile (user-interfaces): Added hurd/tioctl.
(note the date: 1996!), the POSIX-mandated fallback
execv(<shell path>, { argv[0], file, argv[1], ..., NULL })
is not being done. Instead, the following is done:
execv(<shell path>, { <shell path>, file, argv[1], ..., NULL })
In other words, the original argv[0] is not preserved, but is replaced by <shell path>. (Look for _PATH_BSHELL in said historical glibc commit, and also in today's glibc file "posix/execvpe.c".)
Aside - what a funny filename, since execvpe() is a glibc extension not in POSIX ;) I've now filed a bug report to the Austin Group: https://www.austingroupbugs.net/view.php?id=1645 And while researching for that, I found that FreeBSD had the same bug up until 2020: https://cgit.freebsd.org/src/commit/?id=301cb491ea On the other hand, the fact that FreeBSD changed and didn't suffer an immediate backlash of breaking programs means that glibc might consider such a change, despite the length of the history it would be changing. So I'll also file a bug to glibc as part of my efforts. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Wed, Mar 22, 2023 at 03:07:01PM -0500, Eric Blake wrote:
On Wed, Mar 22, 2023 at 04:53:42PM +0100, Laszlo Ersek wrote:
The solution is that glibc *too* has a bug, and that bug hides the busybox bug. Namely, in glibc, going back to historical commit
commit 6a032d81581978187f562e5533a32e0a6a3d352b (tag: cvs/libc-960210)
In other words, the original argv[0] is not preserved, but is replaced by <shell path>. (Look for _PATH_BSHELL in said historical glibc commit, and also in today's glibc file "posix/execvpe.c".)
Aside - what a funny filename, since execvpe() is a glibc extension not in POSIX ;)
I've now filed a bug report to the Austin Group: https://www.austingroupbugs.net/view.php?id=1645 And while researching for that, I found that FreeBSD had the same bug up until 2020: https://cgit.freebsd.org/src/commit/?id=301cb491ea
On the other hand, the fact that FreeBSD changed and didn't suffer an immediate backlash of breaking programs means that glibc might consider such a change, despite the length of the history it would be changing. So I'll also file a bug to glibc as part of my efforts.
The glibc bug is now filed: https://sourceware.org/bugzilla/show_bug.cgi?id=30262 And while doing all this, I also spotted another (unrelated) bug in POSIX: Issue 8 will be moving to C17, which introduces at_quick_exit(); the exec page needs an update to match that (hmm - the draft 3 document I submitted this against is not yet public, but it's coming Real Soon Now (tm)). https://www.austingroupbugs.net/view.php?id=1646 (And I've been using C99 for so long that I was unaware of C17's at_quick_exit() before this week) What a busy day, filing bugs against multiple projects! -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Wed, Mar 22, 2023 at 03:45:17PM +0100, Laszlo Ersek wrote:
On 3/21/23 18:28, Eric Blake wrote:
it is indeed a bug in busybox now that POSIX is moving towards standardizing realpath, so I've filed it: https://bugs.busybox.net/show_bug.cgi?id=15466
I've found another busybox bug.
The "/bin/sh" utility is provided by busybox as well (via the usual symlinking).
Per POSIX, if
execvp(file, { argv[0], argv[1], ..., NULL })
were to fail with -1/ENOEXEC, then execvp() must retry "as if" with
execv(<shell path>, { argv[0], file, argv[1], ..., NULL })
In other words, if direct execution of "file" failed because "file" "has the appropriate access permission but has an unrecognized format", then execvp() is required to try executing "file" as a shell script. For that, <shell path> is left unspecified by POSIX, but the arguments of the shell are specified:
- Argv[0] remains the same. That is, what we wanted "file" to know itself as, is what we now want *the shell executable* to know itself as.
- argv[1] becomes "file" -- this is the script that the shell is supposed to run.
- argv[2] and onwards become positional parameters $1, $2, ... for the shell script.
And the argv[0] specification is what's violated by busybox, because if argv[0] is anything other than "sh", then the busybox binary doesn't recognize itself as the shell!
The as-if rule might allow us to invoke something like execv(<shell path>, {"sh", "-c", munge(file), argv[0], argv[1], ..., NULL}, where munge(file) produces ". quoted_file" as a way to source the contents of file in the current shell environment, without ';' or other metacharacters in file causing us to go off the rail. But getting munge(file) to work correctly without post-fork() malloc() is going to be just as difficult (our _init function pre-fork would have to pre-munge every candidate name...) The busybox list is annoying - it won't let me post without first being a subscriber (I attempted to post a quick patch to implement 'readlink -- foo'; implementing 'realpath -- foo' was not as quick). But if they let me on the list, I'll certainly bring it to their attention that their 'sh' behavior is indeed awkward.
The simplest way to demonstrate the bug is this:
bash-5.2$ ( exec -a foobar /bin/sh <<< "echo hello" ) foobar: applet not found
And then, another way to demonstrate the same busybox issue... lets us, in fact, discover a musl bug in turn!!!
I'll discuss that more in reply to your followup mail. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On 3/22/23 17:10, Eric Blake wrote:
On Wed, Mar 22, 2023 at 03:45:17PM +0100, Laszlo Ersek wrote:
On 3/21/23 18:28, Eric Blake wrote:
it is indeed a bug in busybox now that POSIX is moving towards standardizing realpath, so I've filed it: https://bugs.busybox.net/show_bug.cgi?id=15466
I've found another busybox bug.
The "/bin/sh" utility is provided by busybox as well (via the usual symlinking).
Per POSIX, if
execvp(file, { argv[0], argv[1], ..., NULL })
were to fail with -1/ENOEXEC, then execvp() must retry "as if" with
execv(<shell path>, { argv[0], file, argv[1], ..., NULL })
In other words, if direct execution of "file" failed because "file" "has the appropriate access permission but has an unrecognized format", then execvp() is required to try executing "file" as a shell script. For that, <shell path> is left unspecified by POSIX, but the arguments of the shell are specified:
- Argv[0] remains the same. That is, what we wanted "file" to know itself as, is what we now want *the shell executable* to know itself as.
- argv[1] becomes "file" -- this is the script that the shell is supposed to run.
- argv[2] and onwards become positional parameters $1, $2, ... for the shell script.
And the argv[0] specification is what's violated by busybox, because if argv[0] is anything other than "sh", then the busybox binary doesn't recognize itself as the shell!
The as-if rule might allow us to invoke something like execv(<shell path>, {"sh", "-c", munge(file), argv[0], argv[1], ..., NULL}, where munge(file) produces ". quoted_file" as a way to source the contents of file in the current shell environment, without ';' or other metacharacters in file causing us to go off the rail. But getting munge(file) to work correctly without post-fork() malloc() is going to be just as difficult (our _init function pre-fork would have to pre-munge every candidate name...)
The busybox list is annoying - it won't let me post without first being a subscriber (I attempted to post a quick patch to implement 'readlink -- foo'; implementing 'realpath -- foo' was not as quick). But if they let me on the list, I'll certainly bring it to their attention that their 'sh' behavior is indeed awkward.
Following your lead (#15466), meanwhile I've filed <https://bugs.busybox.net/show_bug.cgi?id=15481>. Thanks! Laszlo
The simplest way to demonstrate the bug is this:
bash-5.2$ ( exec -a foobar /bin/sh <<< "echo hello" ) foobar: applet not found
And then, another way to demonstrate the same busybox issue... lets us, in fact, discover a musl bug in turn!!!
I'll discuss that more in reply to your followup mail.
On Tue, Mar 21, 2023 at 03:56:22PM +0100, Laszlo Ersek wrote:
On 3/21/23 15:05, Eric Blake wrote:
On Tue, Mar 21, 2023 at 07:04:59AM +0100, Laszlo Ersek wrote:
On 3/20/23 20:41, Eric Blake wrote:
On Sun, Mar 19, 2023 at 10:41:37AM +0100, Laszlo Ersek wrote:
This is version 4 of the following sub-series:
[libnbd PATCH v3 09/29] lib/utils: introduce async-signal-safe execvpe() [libnbd PATCH v3 10/29] lib/utils: add unit tests for async-signal-safe execvpe()
...
Series merged as commit range 742cbd8c7adc..0b7172b3cffa.
I see you already fixed one pipeline failure due to some gcc versions being more picky about function __attribute__ placement than others. The remaining failures are with alpine Linux, where /bin/expr comes from busybox instead of coreutils, and has the unfortunate quality of having its behavior dependent on argv[0].
:)
I just wanted to announce on-list that I disabled "lib/test-fork-safe-execvpe.sh" on Alpine Linux because of this :)
Starting from a clean clone, I reproduced it locally with:
$ git diff diff --git i/lib/test-fork-safe-execvpe.sh w/lib/test-fork-safe-execvpe.sh index 838bac9..4b3700c 100755 --- i/lib/test-fork-safe-execvpe.sh +++ w/lib/test-fork-safe-execvpe.sh @@ -18,7 +18,7 @@
. ../tests/functions.sh
-set -e +set -ex
# Determine the absolute pathname of the execvpe helper binary. The "realpath" # utility is not in POSIX, but Linux, FreeBSD and OpenBSD all have it. @@ -155,7 +155,7 @@ success()
# Create a temporary directory and change the working directory to it. tmpd=$(mktemp -d) -cleanup_fn rm -r -- "$tmpd" +#cleanup_fn rm -r -- "$tmpd" cd "$tmpd"
# If the "file" parameter of execvpe() is an empty string, then we must fail --
$ podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge $ podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo libnbd-alpine-edge bash $ ./configure $ make check $ grep tmpd= lib/test-suite.log + tmpd=/tmp/tmp.EMgKeF $ /tmp/tmp.EMgKeF/bin/f 1 + 1 f: applet not found 0b748c9fe495:~$
So it looks like we need some way to work around busybox' insistance that argv[0] determines which applet to run.
I couldn't come up with a reproducer like yours. I couldn't figure out how to *quickly* get an interactive Alpine Linux environment with the test failing, and I also couldn't figure out how to trigger "verbose" test runs on gitlab, without polluting the master branch with debug patches. (I tried forking the project in my own space on gitlab, and pushed a debug patch with just "set -x" onto a non-master branch *there* -- but CI didn't start in response to that.)
So, I only installed a new Alpine Linux virtual machine -- what a pain *that* was --, and investigated what /usr/bin/expr was. When I found it was a symlink to /bin/busybox, I started looking for Alpine Linux specific tweaks that could replace busybox (in this role) with a real binary executable "expr" utility.
I was relieved to find the following wiki article:
https://wiki.alpinelinux.org/wiki/How_to_get_regular_stuff_working
which promised -- I thought anyways -- a real coreutils package.
Imagine my dismay when I found that, after installing coreutils with apk in the Alpine Linux VM, the symlink stayed in place, only its target binary changed from "/bin/busybox" to "coreutils". Well done, Alpine Linux, well done.
So, no, this mess (= Alpine Linux) is not salvageable. The "lib/test-fork-safe-execvpe.sh" script depends on "expr" being functional under the name "f". And yes I want "f" to be a single-character filename; otherwise the nice tabular formatting of the script falls apart (or produces overlong lines).
As last step, I learned about ci/skipped_tests, and used it to disable the test on alpine linux.
The latest pipeline passed: <https://gitlab.com/nbdkit/libnbd/-/pipelines/813280321>.
Either way, I wanted to highlight the following commits on the list:
1 b29ff42e5d00 lib: account for realpath deficiency on some platforms 2 65631e5dfff5 lib/utils: try to placate attribute placement restriction from gcc 3 4cae20ccefaf Revert "lib: account for realpath deficiency on some platforms" 4 f5a065aa3a9c ci: skip "lib/test-fork-safe-execvpe.sh" on Alpine Linux
IIUC, that skips the test in CI, but if a developer or downstream user runs 'make check' won't they still execute test-fork-safe-execvpe.sh ? IOW, will the next release of libnbd have a broken test when it gets imported into the Alpine distro ? FWIW, in terms of CI I'd suggest that skipping Alpine is something that is an especially bad idea. Alpine is the only distro in CI that uses Musl instead of GLibc, and Alpine also does other relatively unusual things compared to other Linux distros, such as the use of busybox that impacted us here. Thus despite the pain, testing on Alpine has a pretty significant value-add for test scenario coverage, compared to testing on lots of other Linux distros which are effectively just different versions of the same regular software setup. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 3/22/23 11:24, Daniel P. Berrangé wrote:
On Tue, Mar 21, 2023 at 03:56:22PM +0100, Laszlo Ersek wrote:
On 3/21/23 15:05, Eric Blake wrote:
On Tue, Mar 21, 2023 at 07:04:59AM +0100, Laszlo Ersek wrote:
On 3/20/23 20:41, Eric Blake wrote:
On Sun, Mar 19, 2023 at 10:41:37AM +0100, Laszlo Ersek wrote:
This is version 4 of the following sub-series:
[libnbd PATCH v3 09/29] lib/utils: introduce async-signal-safe execvpe() [libnbd PATCH v3 10/29] lib/utils: add unit tests for async-signal-safe execvpe()
...
Series merged as commit range 742cbd8c7adc..0b7172b3cffa.
I see you already fixed one pipeline failure due to some gcc versions being more picky about function __attribute__ placement than others. The remaining failures are with alpine Linux, where /bin/expr comes from busybox instead of coreutils, and has the unfortunate quality of having its behavior dependent on argv[0].
:)
I just wanted to announce on-list that I disabled "lib/test-fork-safe-execvpe.sh" on Alpine Linux because of this :)
Starting from a clean clone, I reproduced it locally with:
$ git diff diff --git i/lib/test-fork-safe-execvpe.sh w/lib/test-fork-safe-execvpe.sh index 838bac9..4b3700c 100755 --- i/lib/test-fork-safe-execvpe.sh +++ w/lib/test-fork-safe-execvpe.sh @@ -18,7 +18,7 @@
. ../tests/functions.sh
-set -e +set -ex
# Determine the absolute pathname of the execvpe helper binary. The "realpath" # utility is not in POSIX, but Linux, FreeBSD and OpenBSD all have it. @@ -155,7 +155,7 @@ success()
# Create a temporary directory and change the working directory to it. tmpd=$(mktemp -d) -cleanup_fn rm -r -- "$tmpd" +#cleanup_fn rm -r -- "$tmpd" cd "$tmpd"
# If the "file" parameter of execvpe() is an empty string, then we must fail --
$ podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge $ podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo libnbd-alpine-edge bash $ ./configure $ make check $ grep tmpd= lib/test-suite.log + tmpd=/tmp/tmp.EMgKeF $ /tmp/tmp.EMgKeF/bin/f 1 + 1 f: applet not found 0b748c9fe495:~$
So it looks like we need some way to work around busybox' insistance that argv[0] determines which applet to run.
I couldn't come up with a reproducer like yours. I couldn't figure out how to *quickly* get an interactive Alpine Linux environment with the test failing, and I also couldn't figure out how to trigger "verbose" test runs on gitlab, without polluting the master branch with debug patches. (I tried forking the project in my own space on gitlab, and pushed a debug patch with just "set -x" onto a non-master branch *there* -- but CI didn't start in response to that.)
So, I only installed a new Alpine Linux virtual machine -- what a pain *that* was --, and investigated what /usr/bin/expr was. When I found it was a symlink to /bin/busybox, I started looking for Alpine Linux specific tweaks that could replace busybox (in this role) with a real binary executable "expr" utility.
I was relieved to find the following wiki article:
https://wiki.alpinelinux.org/wiki/How_to_get_regular_stuff_working
which promised -- I thought anyways -- a real coreutils package.
Imagine my dismay when I found that, after installing coreutils with apk in the Alpine Linux VM, the symlink stayed in place, only its target binary changed from "/bin/busybox" to "coreutils". Well done, Alpine Linux, well done.
So, no, this mess (= Alpine Linux) is not salvageable. The "lib/test-fork-safe-execvpe.sh" script depends on "expr" being functional under the name "f". And yes I want "f" to be a single-character filename; otherwise the nice tabular formatting of the script falls apart (or produces overlong lines).
As last step, I learned about ci/skipped_tests, and used it to disable the test on alpine linux.
The latest pipeline passed: <https://gitlab.com/nbdkit/libnbd/-/pipelines/813280321>.
Either way, I wanted to highlight the following commits on the list:
1 b29ff42e5d00 lib: account for realpath deficiency on some platforms 2 65631e5dfff5 lib/utils: try to placate attribute placement restriction from gcc 3 4cae20ccefaf Revert "lib: account for realpath deficiency on some platforms" 4 f5a065aa3a9c ci: skip "lib/test-fork-safe-execvpe.sh" on Alpine Linux
IIUC, that skips the test in CI, but if a developer or downstream user runs 'make check' won't they still execute test-fork-safe-execvpe.sh ? IOW, will the next release of libnbd have a broken test when it gets imported into the Alpine distro ?
FWIW, in terms of CI I'd suggest that skipping Alpine is something that is an especially bad idea. Alpine is the only distro in CI that uses Musl instead of GLibc, and Alpine also does other relatively unusual things compared to other Linux distros, such as the use of busybox that impacted us here. Thus despite the pain, testing on Alpine has a pretty significant value-add for test scenario coverage, compared to testing on lots of other Linux distros which are effectively just different versions of the same regular software setup.
Right, I'm working on this. Still struggling with the podman setup. The UID issue has been sorted (with Giuseppe's help -- I had a very old binary DB laying around). Now the "podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge" command is failing with a different error message -- the download completes, but the internal relinking etc fails due to permission errors, which I don't understand. I've asked Martin for comments. Meanwhile, your other email (= just download the prebuilt container from gitlab) could help! Thanks! Laszlo
On 3/22/23 11:42, Laszlo Ersek wrote:
Now the "podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge" command is failing with a different error message -- the download completes, but the internal relinking etc fails due to permission errors, which I don't understand. I've asked Martin for comments.
Meanwhile, your other email (= just download the prebuilt container from gitlab) could help!
Unfortunately, I got the same failure: podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo \ registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest \ bash
Trying to pull registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest... Getting image source signatures Copying blob 88ecf269dec3 done Copying blob 0ded2f83af0e done Copying config a3b4bffb18 done Writing manifest to image destination Storing signatures Error relocating /usr/lib/libreadline.so.8: RELRO protection failed: Permission denied Error relocating /lib/ld-musl-x86_64.so.1: RELRO protection failed: Permission denied Error relocating /usr/lib/libncursesw.so.6: RELRO protection failed: Permission denied Error relocating /bin/bash: RELRO protection failed: Permission denied
I'm doing this as UID 1000 on RHEL-9.1. Laszlo
On Wed, Mar 22, 2023 at 12:13:49PM +0100, Laszlo Ersek wrote:
On 3/22/23 11:42, Laszlo Ersek wrote:
Now the "podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge" command is failing with a different error message -- the download completes, but the internal relinking etc fails due to permission errors, which I don't understand. I've asked Martin for comments.
Meanwhile, your other email (= just download the prebuilt container from gitlab) could help!
Unfortunately, I got the same failure:
podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo \ registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest \ bash
Trying to pull registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest... Getting image source signatures Copying blob 88ecf269dec3 done Copying blob 0ded2f83af0e done Copying config a3b4bffb18 done Writing manifest to image destination Storing signatures Error relocating /usr/lib/libreadline.so.8: RELRO protection failed: Permission denied Error relocating /lib/ld-musl-x86_64.so.1: RELRO protection failed: Permission denied Error relocating /usr/lib/libncursesw.so.6: RELRO protection failed: Permission denied Error relocating /bin/bash: RELRO protection failed: Permission denied
This looks relevant: https://bugzilla.redhat.com/show_bug.cgi?id=2019324 and suggests restorecon -R ~/.local/share/containers/storage/overlay* With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 3/22/23 12:42, Daniel P. Berrangé wrote:
On Wed, Mar 22, 2023 at 12:13:49PM +0100, Laszlo Ersek wrote:
On 3/22/23 11:42, Laszlo Ersek wrote:
Now the "podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge" command is failing with a different error message -- the download completes, but the internal relinking etc fails due to permission errors, which I don't understand. I've asked Martin for comments.
Meanwhile, your other email (= just download the prebuilt container from gitlab) could help!
Unfortunately, I got the same failure:
podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo \ registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest \ bash
Trying to pull registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest... Getting image source signatures Copying blob 88ecf269dec3 done Copying blob 0ded2f83af0e done Copying config a3b4bffb18 done Writing manifest to image destination Storing signatures Error relocating /usr/lib/libreadline.so.8: RELRO protection failed: Permission denied Error relocating /lib/ld-musl-x86_64.so.1: RELRO protection failed: Permission denied Error relocating /usr/lib/libncursesw.so.6: RELRO protection failed: Permission denied Error relocating /bin/bash: RELRO protection failed: Permission denied
This looks relevant:
https://bugzilla.redhat.com/show_bug.cgi?id=2019324
and suggests
restorecon -R ~/.local/share/containers/storage/overlay*
Yes, I've tried that (via some other links); it does not help. (In the first place, I started with a nonexistent ~/.local/share/containers directory, so I'm unsure why I'm responsible for setting the labels on new contents... but anyway, I tried it and it does not help.) I'll check with setenforce 0 next... Thanks! Laszlo
On 3/22/23 12:45, Laszlo Ersek wrote:
On 3/22/23 12:42, Daniel P. Berrangé wrote:
On Wed, Mar 22, 2023 at 12:13:49PM +0100, Laszlo Ersek wrote:
On 3/22/23 11:42, Laszlo Ersek wrote:
Now the "podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge" command is failing with a different error message -- the download completes, but the internal relinking etc fails due to permission errors, which I don't understand. I've asked Martin for comments.
Meanwhile, your other email (= just download the prebuilt container from gitlab) could help!
Unfortunately, I got the same failure:
podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo \ registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest \ bash
Trying to pull registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest... Getting image source signatures Copying blob 88ecf269dec3 done Copying blob 0ded2f83af0e done Copying config a3b4bffb18 done Writing manifest to image destination Storing signatures Error relocating /usr/lib/libreadline.so.8: RELRO protection failed: Permission denied Error relocating /lib/ld-musl-x86_64.so.1: RELRO protection failed: Permission denied Error relocating /usr/lib/libncursesw.so.6: RELRO protection failed: Permission denied Error relocating /bin/bash: RELRO protection failed: Permission denied
This looks relevant:
https://bugzilla.redhat.com/show_bug.cgi?id=2019324
and suggests
restorecon -R ~/.local/share/containers/storage/overlay*
Yes, I've tried that (via some other links); it does not help. (In the first place, I started with a nonexistent ~/.local/share/containers directory, so I'm unsure why I'm responsible for setting the labels on new contents... but anyway, I tried it and it does not help.) I'll check with setenforce 0 next...
This seems to be a RHEL-9.1 SELinux bug alright. The system is an up-to-date RHEL-9.1 install. (1) I removed the ~/.local/share/containers directory recursively, set SELinux to Permissive mode, and repeated the above podman command. The container was entered alright, and one AVC was logged. Sealert said:
SELinux is preventing /bin/bash from read access on the file /usr/lib/libreadline.so.8.2.
***** Plugin restorecon (99.5 confidence) suggests ************************
If you want to fix the label. /usr/lib/libreadline.so.8.2 default label should be lib_t. Then you can run restorecon. The access attempt may have been stopped due to insufficient permissions to access a parent directory in which case try to change the following command accordingly. Do # /sbin/restorecon -v /usr/lib/libreadline.so.8.2
***** Plugin catchall (1.49 confidence) suggests **************************
If you believe that bash should be allowed read access on the libreadline.so.8.2 file by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'bash' --raw | audit2allow -M my-bash # semodule -X 300 -i my-bash.pp
Additional Information: Source Context system_u:system_r:container_t:s0:c62,c364 Target Context unconfined_u:object_r:user_home_t:s0 Target Objects /usr/lib/libreadline.so.8.2 [ file ] Source bash Source Path /bin/bash Port <Unknown> Host <Unknown> Source RPM Packages bash-5.1.8-6.el9_1.x86_64 Target RPM Packages SELinux Policy RPM selinux-policy-targeted-34.1.43-1.el9_1.2.noarch Local Policy RPM selinux-policy-targeted-34.1.43-1.el9_1.2.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Permissive Host Name lacos-laptop-9.usersys.redhat.com Platform Linux lacos-laptop-9.usersys.redhat.com 5.14.0-162.18.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 9 04:28:41 EST 2023 x86_64 x86_64 Alert Count 1 First Seen 2023-03-22 12:57:44 CET Last Seen 2023-03-22 12:57:44 CET Local ID 0db129a5-552f-49b2-b3bc-ec206978affb
Raw Audit Messages type=AVC msg=audit(1679486264.987:145): avc: denied { read } for pid=2752 comm="bash" path="/usr/lib/libreadline.so.8.2" dev="dm-3" ino=2907654 scontext=system_u:system_r:container_t:s0:c62,c364 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1
type=SYSCALL msg=audit(1679486264.987:145): arch=x86_64 syscall=mprotect success=yes exit=0 a0=7f761e694000 a1=3000 a2=1 a3=55744feb9c80 items=0 ppid=2749 pid=2752 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=2 comm=bash exe=/bin/bash subj=system_u:system_r:container_t:s0:c62,c364 key=(null)ARCH=x86_64 SYSCALL=mprotect AUID=lacos UID=lacos GID=lacos EUID=lacos SUID=lacos FSUID=lacos EGID=lacos SGID=lacos FSGID=lacos
Hash: bash,container_t,user_home_t,file,read
Any comments about "/usr/lib/libreadline.so.8.2" having a bad label are bogus, that file exists within the container image! (2) I ran "restorecon -FvvR ~/.local/share/containers/", and it relabeled a whole bunch of files. Then I repeated the same podman command. The container was entered again, but an effectively identical AVC was logged again. It's easier to show the diff:
@@ -1,5 +1,5 @@
-found 1 alerts in /home/lacos/tmp/1 +found 1 alerts in /home/lacos/tmp/2 --------------------------------------------------------------------------------
SELinux is preventing /bin/bash from read access on the file /usr/lib/libreadline.so.8.2. @@ -24,7 +24,7 @@
Additional Information: -Source Context system_u:system_r:container_t:s0:c62,c364 +Source Context system_u:system_r:container_t:s0:c436,c873 Target Context unconfined_u:object_r:user_home_t:s0 Target Objects /usr/lib/libreadline.so.8.2 [ file ] Source bash @@ -44,15 +44,15 @@ PREEMPT_DYNAMIC Thu Feb 9 04:28:41 EST 2023 x86_64 x86_64 Alert Count 1 -First Seen 2023-03-22 12:57:44 CET -Last Seen 2023-03-22 12:57:44 CET -Local ID 0db129a5-552f-49b2-b3bc-ec206978affb +First Seen 2023-03-22 13:01:49 CET +Last Seen 2023-03-22 13:01:49 CET +Local ID 2771711b-e2af-4c92-840d-36573a4fb12a
Raw Audit Messages -type=AVC msg=audit(1679486264.987:145): avc: denied { read } for pid=2752 comm="bash" path="/usr/lib/libreadline.so.8.2" dev="dm-3" ino=2907654 scontext=system_u:system_r:container_t:s0:c62,c364 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1 +type=AVC msg=audit(1679486509.713:167): avc: denied { read } for pid=3168 comm="bash" path="/usr/lib/libreadline.so.8.2" dev="dm-3" ino=2907654 scontext=system_u:system_r:container_t:s0:c436,c873 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1
-type=SYSCALL msg=audit(1679486264.987:145): arch=x86_64 syscall=mprotect success=yes exit=0 a0=7f761e694000 a1=3000 a2=1 a3=55744feb9c80 items=0 ppid=2749 pid=2752 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=2 comm=bash exe=/bin/bash subj=system_u:system_r:container_t:s0:c62,c364 key=(null)ARCH=x86_64 SYSCALL=mprotect AUID=lacos UID=lacos GID=lacos EUID=lacos SUID=lacos FSUID=lacos EGID=lacos SGID=lacos FSGID=lacos +type=SYSCALL msg=audit(1679486509.713:167): arch=x86_64 syscall=mprotect success=yes exit=0 a0=7f6318db1000 a1=3000 a2=1 a3=562c3fdd6c80 items=0 ppid=3165 pid=3168 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=2 comm=bash exe=/bin/bash subj=system_u:system_r:container_t:s0:c436,c873 key=(null)ARCH=x86_64 SYSCALL=mprotect AUID=lacos UID=lacos GID=lacos EUID=lacos SUID=lacos FSUID=lacos EGID=lacos SGID=lacos FSGID=lacos
Hash: bash,container_t,user_home_t,file,read
Laszlo
On 3/22/23 13:30, Laszlo Ersek wrote:
This seems to be a RHEL-9.1 SELinux bug alright. The system is an up-to-date RHEL-9.1 install.
Filed as <https://bugzilla.redhat.com/show_bug.cgi?id=2181008>. Laszlo
On Tue, Mar 21, 2023 at 09:05:12AM -0500, Eric Blake wrote:
$ podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge
You usually shouldn't need this step when reproducing a problem on CI. Instead just use the container that gitlab has published under the project's registry For any job eg https://gitlab.com/nbdkit/libnbd/-/jobs/3973002673 you'll see at about line 7/8 in the log a line like Using Docker executor with image registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest That image URL is accessible remotely on your local dev machine eg podman run -it registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest You should only need to do 'podman build' if you're debugging a suspected problem with the container image itself.
$ podman run -it --rm --userns=keep-id -v .:/repo:z -w /repo libnbd-alpine-edge bash $ ./configure $ make check $ grep tmpd= lib/test-suite.log + tmpd=/tmp/tmp.EMgKeF $ /tmp/tmp.EMgKeF/bin/f 1 + 1 f: applet not found 0b748c9fe495:~$
So it looks like we need some way to work around busybox' insistance that argv[0] determines which applet to run.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Mar 22, 2023 at 10:13:10AM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 21, 2023 at 09:05:12AM -0500, Eric Blake wrote:
$ podman build -f ci/containers/alpine-edge.Dockerfile -t libnbd-alpine-edge
You usually shouldn't need this step when reproducing a problem on CI. Instead just use the container that gitlab has published under the project's registry
For any job eg
https://gitlab.com/nbdkit/libnbd/-/jobs/3973002673
you'll see at about line 7/8 in the log a line like
Using Docker executor with image registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest
That image URL is accessible remotely on your local dev machine eg
podman run -it registry.gitlab.com/nbdkit/libnbd/ci-alpine-edge:latest
You should only need to do 'podman build' if you're debugging a suspected problem with the container image itself.
Useful to know; we should probably update ci/README.rst accordingly. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
participants (3)
-
Daniel P. Berrangé -
Eric Blake -
Laszlo Ersek