March 2016 - Libguestfs - libguestfs List Archives

[PATCH] inspect: use os-release for CoreOS
by Pino Toscano 31 Mar '16

31 Mar '16

Look for /lib/os-release in the /usr partition and try to use it, if present, before using lsb-release later on. This should not change the final result of the inspection, while using the os-release detection method also for CoreOS. --- Nikos, since you added the support for CoreOS in libguestfs, can you please check whether this change works for you as well? Thanks in advance. src/inspect-fs-unix.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/inspect-fs-unix.c b/src/inspect-fs-unix.c index 35181d6..dab4370 100644 --- a/src/inspect-fs-unix.c +++ b/src/inspect-fs-unix.c @@ -196,6 +196,8 @@ parse_os_release (guestfs_h *g, struct inspect_fs *fs, const char *filename) distro = OS_DISTRO_ARCHLINUX; else if (VALUE_IS ("centos")) distro = OS_DISTRO_CENTOS; + else if (VALUE_IS ("coreos")) + distro = OS_DISTRO_COREOS; else if (VALUE_IS ("debian")) distro = OS_DISTRO_DEBIAN; else if (VALUE_IS ("fedora")) @@ -1055,6 +1057,16 @@ guestfs_int_check_coreos_usr (guestfs_h *g, struct inspect_fs *fs) fs->type = OS_TYPE_LINUX; fs->distro = OS_DISTRO_COREOS; + + if (guestfs_is_file_opts (g, "/lib/os-release", + GUESTFS_IS_FILE_OPTS_FOLLOWSYMLINKS, 1, -1) > 0) { + r = parse_os_release (g, fs, "/lib/os-release"); + if (r == -1) /* error */ + return -1; + if (r == 1) /* ok - detected the release from this file */ + goto skip_release_checks; + } + if (guestfs_is_file_opts (g, "/share/coreos/lsb-release", GUESTFS_IS_FILE_OPTS_FOLLOWSYMLINKS, 1, -1) > 0) { r = parse_lsb_release (g, fs, "/share/coreos/lsb-release"); @@ -1062,6 +1074,8 @@ guestfs_int_check_coreos_usr (guestfs_h *g, struct inspect_fs *fs) return -1; } + skip_release_checks:; + /* Determine the architecture. */ check_architecture (g, fs); -- 2.5.5

3 3

[PATCH 0/3] rename icat API into download_inode
by Matteo Cafasso 31 Mar '16

31 Mar '16

The command name is a bit confusing because it's similar to "cat" but act as "donwload". Therefore I am renaming it with a more clear name. At the same go I cleaned up a bit the code following the standards and improved the API documentation. This patch is ready for review. Code available at: https://github.com/noxdafox/libguestfs/tree/download_inode Matteo Cafasso (3): Rename icat command in download_inode Improve download_inode documentation string. Code cleanup in daemon/sleuthkit.c daemon/sleuthkit.c | 50 ++++++++++++++++++++----------------- generator/actions.ml | 10 +++++--- tests/tsk/Makefile.am | 2 +- tests/tsk/test-download-inode.sh | 53 ++++++++++++++++++++++++++++++++++++++++ tests/tsk/test-icat.sh | 53 ---------------------------------------- 5 files changed, 88 insertions(+), 80 deletions(-) create mode 100755 tests/tsk/test-download-inode.sh delete mode 100755 tests/tsk/test-icat.sh -- 2.8.0.rc3

2 4

virt-v2v become zombie via python cpopen on error
by Shahar Havivi 31 Mar '16

31 Mar '16

Hi, We encounter a problem in VDSM project that virt-v2v become zombie task while importing vm from vmware. When virt-v2v is in 'copy disk' mode and we someone deletes the vm at vmware the process hang in read() method, I am pretty sure that its not virt-v2v problem because when I run it from the shell virt-v2v exit with an error, still maybe someone have an idea.... I wrote a small python script that encounter the problem: ---------------------------------------------------------------------------- from cpopen import CPopen env = {'LIBGUESTFS_BACKEND': 'direct'} cmd = ['/usr/bin/virt-v2v', '-ic', 'vpx://....', '-o', 'local', '-os', '/tmp', '-of', 'raw', '-oa', 'sparse', '--password-file', '/tmp/passwd', '--machine-readable', 'bbb'] p = CPopen(cmd, env=env) while p.returncode is None: c = p.stdout.read(1) print c ---------------------------------------------------------------------------- Thanks, Shahar Havivi.

5 6

[PATCH] renamed daemon/tsk.c to daemon/sleuthkit.c
by Matteo Cafasso 30 Mar '16

30 Mar '16

In order to support the new features I am renaming the file with a better name. The file sleuthkit.c will contain the code depending on the sleuthkit package. The original tsk.c file will contain the logic built using libtsk which is the sleuthkit core library. This patch is ready for review. Code available at: https://github.com/noxdafox/libguestfs/tree/sleuthkit_rename Signed-off-by: Matteo Cafasso <noxdafox(a)gmail.com> --- daemon/Makefile.am | 2 +- daemon/sleuthkit.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++++ daemon/tsk.c | 114 ----------------------------------------------------- 3 files changed, 115 insertions(+), 115 deletions(-) create mode 100644 daemon/sleuthkit.c delete mode 100644 daemon/tsk.c diff --git a/daemon/Makefile.am b/daemon/Makefile.am index 4e2051b..beb7962 100644 --- a/daemon/Makefile.am +++ b/daemon/Makefile.am @@ -171,6 +171,7 @@ guestfsd_SOURCES = \ sfdisk.c \ sh.c \ sleep.c \ + sleuthkit.c \ stat.c \ statvfs.c \ strings.c \ @@ -179,7 +180,6 @@ guestfsd_SOURCES = \ syslinux.c \ tar.c \ truncate.c \ - tsk.c \ umask.c \ upload.c \ utimens.c \ diff --git a/daemon/sleuthkit.c b/daemon/sleuthkit.c new file mode 100644 index 0000000..0fe1250 --- /dev/null +++ b/daemon/sleuthkit.c @@ -0,0 +1,114 @@ +/* libguestfs - the guestfsd daemon + * Copyright (C) 2016 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <inttypes.h> +#include <string.h> +#include <unistd.h> + +#include "guestfs_protocol.h" +#include "daemon.h" +#include "actions.h" +#include "optgroups.h" + +static int file_out (const char *cmd); + +GUESTFSD_EXT_CMD(str_sleuthkit_probe, icat); + +int +optgroup_sleuthkit_available (void) +{ + return prog_exists (str_sleuthkit_probe); +} + +int +do_icat (const mountable_t *mountable, int64_t inode) +{ + CLEANUP_FREE char *cmd = NULL; + + /* Inode must be greater than 0 */ + if (inode < 0) { + reply_with_error ("inode must be >= 0"); + return -1; + } + + /* Construct the command. */ + if (asprintf (&cmd, "icat -r %s %" PRIi64, mountable->device, inode) == -1) { + reply_with_perror ("asprintf"); + return -1; + } + + return file_out (cmd); +} + +static int +file_out (const char *cmd) +{ + int r; + FILE *fp; + CLEANUP_FREE char *buffer = NULL; + + if (verbose) + fprintf (stderr, "%s\n", cmd); + + buffer = malloc (GUESTFS_MAX_CHUNK_SIZE); + if (buffer == NULL) { + reply_with_perror ("malloc"); + return -1; + } + + fp = popen (cmd, "r"); + if (fp == NULL) { + reply_with_perror ("%s", cmd); + return -1; + } + + /* Now we must send the reply message, before the file contents. After + * this there is no opportunity in the protocol to send any error + * message back. Instead we can only cancel the transfer. + */ + reply (NULL, NULL); + + while ((r = fread (buffer, 1, sizeof buffer, fp)) > 0) { + if (send_file_write (buffer, r) < 0) { + pclose (fp); + return -1; + } + } + + if (ferror (fp)) { + fprintf (stderr, "fread: %m"); + send_file_end (1); /* Cancel. */ + pclose (fp); + return -1; + } + + if (pclose (fp) != 0) { + fprintf (stderr, "pclose: %m"); + send_file_end (1); /* Cancel. */ + return -1; + } + + if (send_file_end (0)) /* Normal end of file. */ + return -1; + + return 0; +} diff --git a/daemon/tsk.c b/daemon/tsk.c deleted file mode 100644 index 0fe1250..0000000 --- a/daemon/tsk.c +++ /dev/null @@ -1,114 +0,0 @@ -/* libguestfs - the guestfsd daemon - * Copyright (C) 2016 Red Hat Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - */ - -#include <config.h> - -#include <stdio.h> -#include <stdlib.h> -#include <inttypes.h> -#include <string.h> -#include <unistd.h> - -#include "guestfs_protocol.h" -#include "daemon.h" -#include "actions.h" -#include "optgroups.h" - -static int file_out (const char *cmd); - -GUESTFSD_EXT_CMD(str_sleuthkit_probe, icat); - -int -optgroup_sleuthkit_available (void) -{ - return prog_exists (str_sleuthkit_probe); -} - -int -do_icat (const mountable_t *mountable, int64_t inode) -{ - CLEANUP_FREE char *cmd = NULL; - - /* Inode must be greater than 0 */ - if (inode < 0) { - reply_with_error ("inode must be >= 0"); - return -1; - } - - /* Construct the command. */ - if (asprintf (&cmd, "icat -r %s %" PRIi64, mountable->device, inode) == -1) { - reply_with_perror ("asprintf"); - return -1; - } - - return file_out (cmd); -} - -static int -file_out (const char *cmd) -{ - int r; - FILE *fp; - CLEANUP_FREE char *buffer = NULL; - - if (verbose) - fprintf (stderr, "%s\n", cmd); - - buffer = malloc (GUESTFS_MAX_CHUNK_SIZE); - if (buffer == NULL) { - reply_with_perror ("malloc"); - return -1; - } - - fp = popen (cmd, "r"); - if (fp == NULL) { - reply_with_perror ("%s", cmd); - return -1; - } - - /* Now we must send the reply message, before the file contents. After - * this there is no opportunity in the protocol to send any error - * message back. Instead we can only cancel the transfer. - */ - reply (NULL, NULL); - - while ((r = fread (buffer, 1, sizeof buffer, fp)) > 0) { - if (send_file_write (buffer, r) < 0) { - pclose (fp); - return -1; - } - } - - if (ferror (fp)) { - fprintf (stderr, "fread: %m"); - send_file_end (1); /* Cancel. */ - pclose (fp); - return -1; - } - - if (pclose (fp) != 0) { - fprintf (stderr, "pclose: %m"); - send_file_end (1); /* Cancel. */ - return -1; - } - - if (send_file_end (0)) /* Normal end of file. */ - return -1; - - return 0; -} -- 2.8.0.rc3

3 2

Re: [Libguestfs] [PATCH 0/2] added filesystem_walk0 low level API
by Pino Toscano 30 Mar '16

30 Mar '16

On Wednesday 30 March 2016 13:45:15 NoxDaFox wrote: > Thanks for the comments, just few notes and questions. Just remember to keep the list in the loop. > 2016-03-30 12:24 GMT+03:00 Pino Toscano <ptoscano(a)redhat.com>: > > > Hi, > > > > while reviewing the new series of tsk-related changes, I'm noticing > > common mistakes in all of them: I'll list them below, not > > copy/paste-ing them every review. > > > > (a) please check the coding style, in particular > > - the lack of space between a function name and its following opening > > parenthesis; e.g. do_something(param) -> do_something (param) > > > > I was complying with what said here: > http://libguestfs.org/guestfs-hacking.1.html > > It refers to K&R style which suggests functions defined as > do_something(param). > > I guess the documentation at the link could mention this detail. Indeed it is something that should be changed, since the rest of the code definitely has spaces between functions and opening parenthesis. > - the style of functions, e.g.: > > | int do_something (...) > > needs to be > > | int > > | do_something (...) > > > > Same as above. What I mentioned is the style used all around the code. > - style of function prototypes (everything in a single line) > > > > I'm confused here: should they be all on a single line or should they be > split as in the note above? Prototypes need a single line, while the actual function definitions indented as above. In general, a golden rule that applies whenever patching any software is "follow the existing style", in case there is a single one (or at least one followed in the majority of the codebase). > > (b) please avoid if statements with side effects, as it makes code > > reading and refactoring harder > > > > Will re-factor the logic. > > > > (c) please put explanations and message in the messages of the actual > > commits, instead of in the cover letters sent to the mailing list: > > otherwise, browsing the git history in the future will not give many > > clues about the newly added APIs > > > > I tried to stick to the behaviour I saw in the mailing list. Will fix this > as well. Usually cover letters provide a general overview of what a series of patches does, while the actual commits still provide explanations of what each does. > > (d) please wrap descriptions for actions at 80 columns max. > > > > Going to fix this as well. -- Pino Toscano

1 0

[PATCH 0/2] rename icat API as download_inode
by Matteo Cafasso 30 Mar '16

30 Mar '16

"icat" name comes from the employed command line tool which might be replaced later on with a different implementation. The command name is a bit confusing because it's similar to "cat" but act as "donwload". Therefore I am renaming it with a more clear name. At the same time I cleaned up a bit the code and improved it's readability and code comments. This patch is ready for review. Code available at: https://github.com/noxdafox/libguestfs/tree/download_inode Matteo Cafasso (2): rename icat API to download_inode updated download_inode tests daemon/sleuthkit.c | 79 ++++++++++++++++++++++------------------ generator/actions.ml | 6 ++- tests/tsk/Makefile.am | 2 +- tests/tsk/test-download-inode.sh | 53 +++++++++++++++++++++++++++ tests/tsk/test-icat.sh | 53 --------------------------- 5 files changed, 101 insertions(+), 92 deletions(-) create mode 100755 tests/tsk/test-download-inode.sh delete mode 100755 tests/tsk/test-icat.sh -- 2.8.0.rc3

2 3

[PATCH 0/2] added filesystem_walk API
by Matteo Cafasso 30 Mar '16

30 Mar '16

The filesystem_walk API parses the FS internals of a partition and returns a list of all the files and directories contained within. It list deleted files and directories as well. For each node, it reports its relative path, its inode and its allocation status. This is the end user API for inspecting a disk partition content. The command can handle filenames with special characters. Example Use Cases: * inspect the FS content to see which deleted files could be possibly retrieved. * quickly check the FS content without mounting the disk * get a full mapping inode -> filepath of the disk Matteo Cafasso (2): added filesystem_walk API added filesystem_walk API tests generator/actions.ml | 13 +++ src/Makefile.am | 1 + src/tsk.c | 194 ++++++++++++++++++++++++++++++++++++++ tests/tsk/Makefile.am | 3 +- tests/tsk/test-filesystem-walk.sh | 48 ++++++++++ 5 files changed, 258 insertions(+), 1 deletion(-) create mode 100644 src/tsk.c create mode 100755 tests/tsk/test-filesystem-walk.sh -- 2.8.0.rc3

2 3

[PATCH 0/2] added filesystem_walk0 low level API
by Matteo Cafasso 30 Mar '16

30 Mar '16

The filesystem_walk0 API parses the FS internals of a partition and returns a list of all the files and directories contained within. It list deleted files and directories as well. For each node, it reports its relative path, its inode and its allocation status. The output is serialised in XDR format and written to the given file. The command is similar to The Sleuth Kit "fls -rp <disk_image>" except from the fact that it does not report the node type (file or directory). Matteo Cafasso (2): added filesystem_walk0 API added filesystem_walk0 API tests daemon/Makefile.am | 3 +- daemon/tsk.c | 186 +++++++++++++++++++++++++++++++++++++ generator/actions.ml | 16 ++++ m4/guestfs_daemon.m4 | 8 ++ src/MAX_PROC_NR | 2 +- tests/tsk/Makefile.am | 3 +- tests/tsk/test-filesystem-walk0.sh | 54 +++++++++++ 7 files changed, 269 insertions(+), 3 deletions(-) create mode 100755 tests/tsk/test-filesystem-walk0.sh -- 2.8.0.rc3

2 4

[PATCH] appliance: Copy /etc/machine-id from host system into the appliance.
by Richard W.M. Jones 29 Mar '16

29 Mar '16

Currently when the appliance is booted we see warning messages like these ones: [/usr/lib/tmpfiles.d/systemd.conf:26] Failed to replace specifiers: /run/log/journal/%m [/usr/lib/tmpfiles.d/systemd.conf:28] Failed to replace specifiers: /run/log/journal/%m [/usr/lib/tmpfiles.d/systemd.conf:29] Failed to replace specifiers: /run/log/journal/%m They are apparently harmless, but are caused because /etc/machine-id does not exist inside the appliance, and so %m cannot be substituted by systemd-tmpfiles. The solution to this is to provide /etc/machine-id. However this is not so simple: either we could get supermin to generate this, requiring an ugly systemd-specific hack to supermin, or we could generate it in the /init script, which just slows the init script down for no real reason. Since the content of /etc/machine-id doesn't really matter for a throwaway appliance so long as it has the right format, this commit copies the host file into the appliance. --- appliance/hostfiles.in | 1 + 1 file changed, 1 insertion(+) diff --git a/appliance/hostfiles.in b/appliance/hostfiles.in index 8ff53b5..6c581bd 100644 --- a/appliance/hostfiles.in +++ b/appliance/hostfiles.in @@ -13,5 +13,6 @@ dnl SUSE=1 For OpenSUSE. dnl FRUGALWARE=1 For Frugalware. dnl MAGEIA=1 For Mageia. +/etc/machine-id /lib/lsb/* /usr/share/augeas/lenses/*.aug -- 2.7.4

3 7

[PATCH 0/4] v2v: more control over device types
by Roman Kagan 25 Mar '16

25 Mar '16

The decision on which device type to use for disks, network and video cards on output used to be taken deep inside the converting functions. This is not always desirable. In particular, there are scenarios when this decision is made before the convertion takes place. E.g. in in-place mode, the decisions are taken and the output VM configuration is created outside of v2v tool. This patchset adds support for such scenarios. Specifically, - the input configuration parsers are taught to extract network and video device models in addition to storage controllers - the converting functions are taught to take the requested device types into consideration - the main routine in in-place mode takes the relevant data from the input VM and passes it to the converter The copying mode may eventually also profit from this patchset by taking command-line options for controlling device types, but this isn't included here. Roman Kagan (4): v2v: collect source network and video adapter types v2v: introduce requested guestcaps type v2v: take requested caps into account when converting v2v: in-place: request caps based on source config test-data/phony-guests/guests.xml.in | 8 +++ v2v/convert_linux.ml | 49 +++++++++++---- v2v/convert_windows.ml | 4 +- v2v/input_disk.ml | 2 + v2v/input_libvirtxml.ml | 32 ++++++++++ v2v/input_ova.ml | 2 + v2v/modules_list.ml | 3 +- v2v/modules_list.mli | 3 +- v2v/test-v2v-i-ova-formats.expected | 1 + v2v/test-v2v-i-ova-gz.expected | 1 + v2v/test-v2v-i-ova-two-disks.expected | 1 + v2v/test-v2v-print-source.sh | 4 +- v2v/types.ml | 73 +++++++++++++++++++---- v2v/types.mli | 17 +++++- v2v/v2v.ml | 58 +++++++++++++++++- v2v/windows_virtio.ml | 108 ++++++++++++++++++++++++---------- v2v/windows_virtio.mli | 8 ++- 17 files changed, 309 insertions(+), 65 deletions(-) -- 2.5.0

3 12