May 2015 - Libguestfs - libguestfs List Archives

[PATCH 0/3] Add support for CoreOS
by Nikos Skalkotos 29 May '15

29 May '15

This patch set adds support for inspecting CoreOS installations. In CoreOS the files that need to be inspected under /etc are links to files under /usr which is hosted on a different partition and mounted read only. Hence, inspecting the root fs is not enough to gather all needed information. This patch set adds code for recognizing the root and the /usr partitions of CoreOS and a new internal API function that will merge the inspection information of the 2 inspect_fs structs into one. Nikos Skalkotos (3): inspection: Add func for merging fs inspections inspection: Add support for CoreOS Add tests for CoreOS .gitignore | 1 + generator/actions.ml | 4 + inspector/Makefile.am | 1 + inspector/expected-coreos.img.xml | 30 +++++++ src/guestfs-internal.h | 4 + src/inspect-fs-unix.c | 69 ++++++++++++++- src/inspect-fs.c | 136 ++++++++++++++++++++++++++++++ src/inspect-icon.c | 1 + src/inspect.c | 62 ++++++++++++++ tests/guests/Makefile.am | 6 ++ tests/guests/guest-aux/make-coreos-img.sh | 83 ++++++++++++++++++ tests/guests/guests.xml.in | 16 ++++ 12 files changed, 410 insertions(+), 3 deletions(-) create mode 100644 inspector/expected-coreos.img.xml create mode 100755 tests/guests/guest-aux/make-coreos-img.sh -- 2.1.0

1 0

[PATCH v2 00/11] virt-resize: add support for resizing MBR logical partitions
by Chen Hanxiao 29 May '15

29 May '15

In current virt-resize, only primary partitions(including extended partition) are supported. They are collected in an array for resize operations. Logical partitions are not supported. This series add support for resizing logical partitions. v2: 1) Add 3 variables to describe relationship of logical and extended partitions: - partitions flat list of primary partitions (as now, the global 'partitions'). extended partitions is essentially primary partition - logical_partitions flat list of logical partitions - extended_partition one MBR extended partition 2) reserve enough size when resizing logical partitions Original patches by Hu Tao at: https://www.redhat.com/archives/libguestfs/2014-October/msg00238.html Chen Hanxiao (11): resize: move loop check from find_partitions resize: add logical_partitions and extended_partition resize: updated find_partition to support logical partition resize: add support for logical partitions for calculate_surplus resize: handle resize of logical partitions resize: add support for logical partitions of calculate_target_partitions resize: calculate_target_partitions for logical partitions resize: parted and copy logical partitions resize: more misc ops on logical partition resize: add support resize extended partition resize: test: add support for resizing extended and logical partitions resize/resize.ml | 189 ++++++++++++++++++++++++++++++++++----------- resize/test-virt-resize.pl | 32 ++------ 2 files changed, 149 insertions(+), 72 deletions(-) -- 2.1.0

4 21

[PATCH 1/4] generator: move api_version to a common version_added
by Pino Toscano 28 May '15

28 May '15

This way the version string of each API can be used also in other generator modules. Mostly code motion, no actual behaviour changes. --- generator/c.ml | 21 ++++++++++----------- generator/docstrings.ml | 6 ++++++ 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/generator/c.ml b/generator/c.ml index 63dc09a..a2b9c94 100644 --- a/generator/c.ml +++ b/generator/c.ml @@ -301,8 +301,10 @@ I<The caller must free the returned buffer after use>.\n\n" pr "This function takes a key or passphrase parameter which could contain sensitive material. Read the section L</KEYS AND PASSPHRASES> for more information.\n\n"; - let version = api_version f.added in - pr "(Added in %s)\n\n" version; + (match version_added f with + | Some version -> pr "(Added in %s)\n\n" version + | None -> assert false + ); (* Handling of optional argument variants. *) if optargs <> [] then ( @@ -322,8 +324,8 @@ L</KEYS AND PASSPHRASES> for more information.\n\n"; pr "See L</CALLS WITH OPTIONAL ARGUMENTS>.\n\n"; ) -and generate_actions_pod_back_compat_entry { name = name; added = added; - style = ret, args, _ } = +and generate_actions_pod_back_compat_entry ({ name = name; + style = ret, args, _ } as f) = pr "=head2 guestfs_%s\n\n" name; generate_prototype ~extern:false ~indent:" " ~handle:"g" ~prefix:"guestfs_" name (ret, args, []); @@ -334,13 +336,10 @@ and generate_actions_pod_back_compat_entry { name = name; added = added; pr "L</guestfs_%s_opts> with no optional arguments.\n" name; pr "\n"; - let version = api_version added in - pr "(Added in %s)\n\n\n" version; - -and api_version = function - | (0, 0, release) -> sprintf "0.%d" release - | ((0|1) as major, minor, release) -> sprintf "%d.%d.%d" major minor release - | _ -> assert false + (match version_added f with + | Some version -> pr "(Added in %s)\n\n\n" version + | None -> assert false + ) and generate_structs_pod () = generate_header PODStyle GPLv2plus; diff --git a/generator/docstrings.ml b/generator/docstrings.ml index 8ca54d1..caf836c 100644 --- a/generator/docstrings.ml +++ b/generator/docstrings.ml @@ -51,6 +51,12 @@ fact that they are deprecated indicates that there are problems with correct use of these functions." prefix alt in Some txt +let version_added = function + | { added = (0, 0, release) } -> Some (sprintf "0.%d" release) + | { added = ((0|1) as major, minor, release) } -> + Some (sprintf "%d.%d.%d" major minor release) + | _ -> None + let copyright_years = let this_year = 1900 + (localtime (time ())).tm_year in if this_year > 2009 then sprintf "2009-%04d" this_year else "2009" -- 2.1.0

2 4

[PATCH v5] inspector: recognize ppc64 and ppc64le archs (RHBZ#1211996)
by Maros Zatko 28 May '15

28 May '15

Patch extracts MSB/LSB info from /bin/file output and passes it as separate parameret from elf_arch. Then it is sent to impl_file_architecture and checked/ Modified magic_for_file to handle regex changes. Now with tests for file_architecture, and error reporting for unknown endianness. Fixes: RHBZ#1211996 Maros Zatko (1): inspector: recognize ppc64 and ppc64le archs (RHBZ#1211996) generator/actions.ml | 8 ++++++++ src/filearch.c | 28 +++++++++++++++++++--------- 2 files changed, 27 insertions(+), 9 deletions(-) -- 1.9.3

2 2

Concurrent scanning of same disk
by NoxDaFox 28 May '15

28 May '15

Greetings, I am suffering of several weird errors which show randomly and make me suspect some concurrency issue. Libguestfs version is 1.28.1, linux kernel 3.16, libvirt 1.2.9 and qemu 2.1. What I'm trying to do is comparing the disk state at two different point of a guest execution. Disk snapshots are taken through libvirt in different moments (I am aware of caching issue), from such snapshots, new disks are created using the "qemu-img convert" command with backing file pointing to the disk being executed. Then, I spawn as many processes as disk images and in each process I create an guestfs instance and mount one of the disks. The operation seems successful as I'm gathering the data I am looking for but I suffer from these random failures: * RuntimeError: file receive cancelled by daemon - On r = libguestfsmod.checksums_out (self._o, csumtype, directory, sumsfile) * RuntimeError: hivex_close: do_hivex_close: you must call 'hivex-open' first to initialize the hivex handle - On r = libguestfsmod.inspect_os (self._o) The random nature of the errors make me think about some concurrency problem. The disk images are mounted read only, each process uses a new guestfs instance and mount a different disk image. Only shared data is the backing file. Same code without backing image was running on a different server. I was able to compare whole disks image created from snapshots of a running one. Most probably I am doing something wrong but I can't figure out what.

3 11

[PATCH RFC] New API: btrfs_convert
by Pino Tsao 27 May '15

27 May '15

Signed-off-by: Pino <caoj.fnst(a)cn.fujitsu.com> --- daemon/btrfs.c | 29 +++++++++++++++++++++++++++++ generator/actions.ml | 18 ++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/daemon/btrfs.c b/daemon/btrfs.c index 39392f7..7f10792 100644 --- a/daemon/btrfs.c +++ b/daemon/btrfs.c @@ -38,6 +38,7 @@ GUESTFSD_EXT_CMD(str_btrfsck, btrfsck); GUESTFSD_EXT_CMD(str_mkfs_btrfs, mkfs.btrfs); GUESTFSD_EXT_CMD(str_umount, umount); GUESTFSD_EXT_CMD(str_btrfsimage, btrfs-image); +GUESTFSD_EXT_CMD(str_btrfsconvert, btrfs-convert); int optgroup_btrfs_available (void) @@ -2083,3 +2084,31 @@ do_btrfs_image (char *const *sources, const char *image, return 0; } + +int +do_btrfs_convert (const char *device, int rollback) +{ + const size_t MAX_ARGS = 64; + const char *argv[MAX_ARGS]; + size_t i = 0; + CLEANUP_FREE char *err = NULL; + CLEANUP_FREE char *out = NULL; + int r; + + ADD_ARG (argv, i, str_btrfsconvert); + ADD_ARG (argv, i, device); + + if ((optargs_bitmask & GUESTFS_BTRFS_CONVERT_ROLLBACK_BITMASK) && rollback) + ADD_ARG (argv, i, "-r"); + + ADD_ARG (argv, i, NULL); + + r = commandv (&out, &err, argv); + if (r == -1) { + reply_with_error ("%s: %s", device, err); + return -1; + } + + return 0; +} + diff --git a/generator/actions.ml b/generator/actions.ml index 1a89869..8c83480 100644 --- a/generator/actions.ml +++ b/generator/actions.ml @@ -12579,6 +12579,24 @@ numbered C<partnum> on device C<device>. It returns C<primary>, C<logical>, or C<extended>." }; + { defaults with + name = "btrfs_convert"; + style = RErr, [Device "device"], [OBool "rollback"]; + proc_nr = Some 455; + optional = Some "btrfs"; camel_name = "BTRFSConvert"; + tests = [ + InitEmpty, Always, TestRun ( + [["mkfs"; "ext2"; "/dev/sda"; ""; "NOARG"; ""; ""; "NOARG"]; + ["btrfs_convert"; "/dev/sda"; ""]; + ["btrfs_convert"; "/dev/sda"; "true"]]), [] + ]; + shortdesc = "convert from ext2/3/4 filesystem to btrfs or rollback"; + longdesc = "\ +This is used to convert existed ext2/3/4 to btrfs filesystem, and the original +filesystem image is accessible as from separate subvolume named ext2_saved as file image. + +If you want to rollback to ext2/3/4, set rollback to true." }; + ] (* Non-API meta-commands available only in guestfish. -- 2.1.0

1 0

[PATCH] lib: Limit space and time used by 'qemu-img info' subprocess.
by Richard W.M. Jones 26 May '15

26 May '15

After fuzzing 'qemu-img info' I found that certain files can cause the command to use lots of memory and time. Modify the command mini-library to allow us to place resource limits on subprocesses, and use these to limit the amount of space and time used by 'qemu-img info'. --- configure.ac | 3 +++ src/command.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++ src/guestfs-internal.h | 1 + src/info.c | 22 ++++++++++++++++++++- 4 files changed, 78 insertions(+), 1 deletion(-) diff --git a/configure.ac b/configure.ac index acd807b..6fea1e4 100644 --- a/configure.ac +++ b/configure.ac @@ -305,8 +305,10 @@ AC_CHECK_HEADERS([\ linux/raid/md_u.h \ printf.h \ sys/inotify.h \ + sys/resource.h \ sys/socket.h \ sys/statvfs.h \ + sys/time.h \ sys/types.h \ sys/un.h \ sys/wait.h \ @@ -335,6 +337,7 @@ AC_CHECK_FUNCS([\ posix_fadvise \ removexattr \ setitimer \ + setrlimit \ setxattr \ sigaction \ statvfs \ diff --git a/src/command.c b/src/command.c index 45ae5d6..993198a 100644 --- a/src/command.c +++ b/src/command.c @@ -77,6 +77,13 @@ #include <sys/wait.h> #include <sys/select.h> +#ifdef HAVE_SYS_TIME_H +#include <sys/time.h> +#endif +#ifdef HAVE_SYS_RESOURCE_H +#include <sys/resource.h> +#endif + #include "guestfs.h" #include "guestfs-internal.h" @@ -101,6 +108,12 @@ struct buffering { void (*close_data) (struct command *cmd); }; +struct child_rlimits { + struct child_rlimits *next; + int resource; + long limit; +}; + struct command { guestfs_h *g; @@ -139,6 +152,9 @@ struct command cmd_child_callback child_callback; void *child_callback_data; + /* Optional child limits. */ + struct child_rlimits *child_rlimits; + /* Optional stdin forwarding to the child. */ int infd; }; @@ -329,6 +345,22 @@ guestfs_int_cmd_set_child_callback (struct command *cmd, cmd->child_callback_data = data; } +/* Set up child rlimits, in case the process we are running could + * consume lots of space or time. + */ +void +guestfs_int_cmd_set_child_rlimit (struct command *cmd, int resource, long limit) +{ + struct child_rlimits *p; + + p = safe_malloc (cmd->g, sizeof *p); + p->resource = resource; + p->limit = limit; + p->next = cmd->child_rlimits; + cmd->child_rlimits = p; +} + + /* Finish off the command by either NULL-terminating the argv array or * adding a terminating \0 to the string, or die with an internal * error if no command has been added. @@ -390,6 +422,10 @@ run_command (struct command *cmd, bool get_stdin_fd, bool get_stdout_fd, int outfd[2] = { -1, -1 }; int infd[2] = { -1, -1 }; char status_string[80]; +#ifdef HAVE_SETRLIMIT + struct child_rlimits *child_rlimit; + struct rlimit rlimit; +#endif get_stdout_fd = get_stdout_fd || cmd->stdout_callback != NULL; get_stderr_fd = get_stderr_fd || cmd->capture_errors; @@ -510,6 +546,23 @@ run_command (struct command *cmd, bool get_stdin_fd, bool get_stdout_fd, _exit (EXIT_FAILURE); } +#ifdef HAVE_SETRLIMIT + for (child_rlimit = cmd->child_rlimits; + child_rlimit != NULL; + child_rlimit = child_rlimit->next) { + rlimit.rlim_cur = rlimit.rlim_max = child_rlimit->limit; + if (setrlimit (child_rlimit->resource, &rlimit) == -1) { + /* EPERM means we're trying to raise the limit (ie. the limit is + * already more restrictive than what we want), so ignore it. + */ + if (errno != EPERM) { + perror ("setrlimit"); + _exit (EXIT_FAILURE); + } + } + } +#endif /* HAVE_SETRLIMIT */ + /* Run the command. */ switch (cmd->style) { case COMMAND_STYLE_EXECV: diff --git a/src/guestfs-internal.h b/src/guestfs-internal.h index 414a634..4f06c37 100644 --- a/src/guestfs-internal.h +++ b/src/guestfs-internal.h @@ -864,6 +864,7 @@ extern void guestfs_int_cmd_set_stdout_callback (struct command *, cmd_stdout_ca #define CMD_STDOUT_FLAG_UNBUFFERED 1 #define CMD_STDOUT_FLAG_WHOLE_BUFFER 2 extern void guestfs_int_cmd_set_stderr_to_stdout (struct command *); +extern void guestfs_int_cmd_set_child_rlimit (struct command *, int resource, long limit); extern void guestfs_int_cmd_clear_capture_errors (struct command *); extern void guestfs_int_cmd_clear_close_files (struct command *); extern void guestfs_int_cmd_set_child_callback (struct command *, cmd_child_callback child_callback, void *data); diff --git a/src/info.c b/src/info.c index bd4221c..bfd7860 100644 --- a/src/info.c +++ b/src/info.c @@ -31,6 +31,14 @@ #include <assert.h> #include <string.h> +#ifdef HAVE_SYS_TIME_H +#include <sys/time.h> +#endif + +#ifdef HAVE_SYS_RESOURCE_H +#include <sys/resource.h> +#endif + #if HAVE_YAJL #include <yajl/yajl_tree.h> #endif @@ -269,7 +277,13 @@ get_json_output (guestfs_h *g, const char *filename) guestfs_int_cmd_add_arg (cmd, "json"); guestfs_int_cmd_add_arg (cmd, fdpath); guestfs_int_cmd_set_stdout_callback (cmd, parse_json, &tree, - CMD_STDOUT_FLAG_WHOLE_BUFFER); + CMD_STDOUT_FLAG_WHOLE_BUFFER); +#ifdef RLIMIT_AS + guestfs_int_cmd_set_child_rlimit (cmd, RLIMIT_AS, 1000000000 /* 1GB */); +#endif +#ifdef RLIMIT_CPU + guestfs_int_cmd_set_child_rlimit (cmd, RLIMIT_CPU, 10 /* seconds */); +#endif r = guestfs_int_cmd_run (cmd); close (fd); if (r == -1) @@ -548,6 +562,12 @@ old_parser_run_qemu_img_info (guestfs_h *g, const char *filename, guestfs_int_cmd_add_arg (cmd, "info"); guestfs_int_cmd_add_arg (cmd, safe_filename); guestfs_int_cmd_set_stdout_callback (cmd, fn, data, 0); +#ifdef RLIMIT_AS + guestfs_int_cmd_set_child_rlimit (cmd, RLIMIT_AS, 1000000000 /* 1GB */); +#endif +#ifdef RLIMIT_CPU + guestfs_int_cmd_set_child_rlimit (cmd, RLIMIT_CPU, 10 /* seconds */); +#endif r = guestfs_int_cmd_run (cmd); if (r == -1) return -1; -- 2.3.1

1 0

[PATCH] api: Don't truncate /dev/stdout or /dev/stderr when used as FileOut.
by Richard W.M. Jones 26 May '15

26 May '15

In APIs such as guestfs_download, when the FileOut parameter exactly matches "/dev/stdout" or "/dev/stderr", don't reopen the possibly redirected output file with O_TRUNC (truncate). Instead dup the file descriptor. This magic behaviour doesn't happen for /dev/fd/* (or any other output file) allowing callers the choice of using /dev/stderr or /dev/fd/2 depending on whether or not they want truncation. This works around an annoying virt-builder bug. If you do: $ virt-builder fedora-21 --install no_such_package -v -x >& /tmp/log then when the `--install' command fails, virt-builder will download the log file using `guestfs_download (g, log, "/dev/stderr")'. Since this truncates the redirected /dev/stderr, the final log file is truncated and corrupted. With this patch the log file is no longer corrupted. --- customize/customize_run.ml | 7 +------ src/proto.c | 13 +++++++++++-- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/customize/customize_run.ml b/customize/customize_run.ml index 5a7e209..d9547a0 100644 --- a/customize/customize_run.ml +++ b/customize/customize_run.ml @@ -45,12 +45,7 @@ let run (g : Guestfs.guestfs) root (ops : ops) = (* Function to cat the log file, for debugging and error messages. *) let debug_logfile () = - try - (* XXX If stderr is redirected this actually truncates the - * redirection file, which is pretty annoying to say the - * least. - *) - g#download logfile "/dev/stderr" + try g#download logfile "/dev/stderr" with exn -> warning (f_"log file %s: %s (ignored)") logfile (Printexc.to_string exn) in diff --git a/src/proto.c b/src/proto.c index a019625..a46a382 100644 --- a/src/proto.c +++ b/src/proto.c @@ -756,9 +756,18 @@ guestfs_int_recv_file (guestfs_h *g, const char *filename) g->user_cancel = 0; - fd = open (filename, O_WRONLY|O_CREAT|O_TRUNC|O_NOCTTY|O_CLOEXEC, 0666); + /* If downloading to /dev/stdout or /dev/stderr, dup the file + * descriptor instead of reopening the file, so that redirected + * stdout/stderr work properly. + */ + if (STREQ (filename, "/dev/stdout")) + fd = dup (1); + else if (STREQ (filename, "/dev/stderr")) + fd = dup (2); + else + fd = open (filename, O_WRONLY|O_CREAT|O_TRUNC|O_NOCTTY|O_CLOEXEC, 0666); if (fd == -1) { - perrorf (g, "open: %s", filename); + perrorf (g, "%s", filename); goto cancel; } -- 2.3.1

2 2

[PATCH 0/6] Update the way that API versions are generated for the man page.
by Richard W.M. Jones 26 May '15

26 May '15

The existing mechanism was clunky, slow and used ~ 10 MB of local disk. Rich.

1 6

supermin in Fedora Rawhide switched (again) to using dnf instead of yum
by Richard W.M. Jones 26 May '15

26 May '15

Previous attempt: https://www.redhat.com/archives/libguestfs/2015-April/msg00013.html supermin needs to download packages (eg. RPMs) when preparing the appliance. After a lot of work, 'dnf download' can now be used as a replacement for the (deprecated) 'yumdownloader' program. This only affects the 'supermin --prepare' phase, which means it only affects people building libguestfs from source. Ordinary use of libguestfs, even in Rawhide, should be unaffected. If you are building libguestfs from source on Fedora please try the new supermin + dnf, and let me know if you find any problems. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

2 4