February 2015 - Libguestfs - libguestfs List Archives

[PATCH] actions.ml: fix a typo
by Chen Hanxiao 13 Feb '15

13 Feb '15

s/scannning/scanning Signed-off-by: Chen Hanxiao <chenhanxiao(a)cn.fujitsu.com> --- generator/actions.ml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/generator/actions.ml b/generator/actions.ml index 5301a29..6c6849d 100644 --- a/generator/actions.ml +++ b/generator/actions.ml @@ -12397,7 +12397,7 @@ Defragment a file or directory on a btrfs filesystem. compress is one of zlib or ]; shortdesc = "recover the chunk tree of btrfs filesystem"; longdesc = "\ -Recover the chunk tree of btrfs filesystem by scannning the devices one by one." }; +Recover the chunk tree of btrfs filesystem by scanning the devices one by one." }; { defaults with name = "btrfs_rescue_super_recover"; -- 2.1.0

2 1

[PATCH 1/3] macosx: Includes/defines for byteswap operations
by Margaret Lewicka 12 Feb '15

12 Feb '15

--- src/inspect-apps.c | 13 ++++++++++++- src/inspect-fs-windows.c | 6 ++++++ src/journal.c | 5 +++++ 3 files changed, 23 insertions(+), 1 deletion(-) diff --git a/src/inspect-apps.c b/src/inspect-apps.c index 20cf00a..8fbae9c 100644 --- a/src/inspect-apps.c +++ b/src/inspect-apps.c @@ -35,11 +35,22 @@ #include <sys/endian.h> #endif -/* be32toh is usually a macro definend in <endian.h>, but it might be +/* be32toh is usually a macro defined in <endian.h>, but it might be * a function in some system so check both, and if neither is defined * then define be32toh for RHEL 5. */ #if !defined(HAVE_BE32TOH) && !defined(be32toh) + +#if defined __APPLE__ && defined __MACH__ +/* Define/include necessary items on MacOS X */ +#include <machine/endian.h> +#define __BIG_ENDIAN BIG_ENDIAN +#define __LITTLE_ENDIAN LITTLE_ENDIAN +#define __BYTE_ORDER BYTE_ORDER +#include <libkern/OSByteOrder.h> +#define __bswap_32 OSSwapConstInt32 +#endif /* __APPLE__ */ + #if __BYTE_ORDER == __LITTLE_ENDIAN #define be32toh(x) __bswap_32 (x) #else diff --git a/src/inspect-fs-windows.c b/src/inspect-fs-windows.c index 682157a..421a5b1 100644 --- a/src/inspect-fs-windows.c +++ b/src/inspect-fs-windows.c @@ -36,6 +36,12 @@ #include <sys/endian.h> #endif +#if defined __APPLE__ && defined __MACH__ +#include <libkern/OSByteOrder.h> +#define le32toh(x) OSSwapLittleToHostInt32(x) +#define le64toh(x) OSSwapLittleToHostInt64(x) +#endif + #include <pcre.h> #include "c-ctype.h" diff --git a/src/journal.c b/src/journal.c index 1070067..c563b7f 100644 --- a/src/journal.c +++ b/src/journal.c @@ -35,6 +35,11 @@ #include <sys/endian.h> #endif +#if defined __APPLE__ && defined __MACH__ +#include <libkern/OSByteOrder.h> +#define be64toh(x) OSSwapBigToHostInt64(x) +#endif + #include "full-read.h" #include "guestfs.h" -- 1.9.3

3 9

[PATCH] macosx: Darwin-specific autoconf macros
by Margaret Lewicka 12 Feb '15

12 Feb '15

* Replace LD_LIBRARY_PATH with DYLD_LIBRARY_PATH for Darwin * Remove the -lcrypt flag for Darwin (unsupported) --- configure.ac | 13 +++++++++++++ run.in | 10 +++++----- v2v/link.sh.in | 2 +- 3 files changed, 19 insertions(+), 6 deletions(-) diff --git a/configure.ac b/configure.ac index d68190a..295de11 100644 --- a/configure.ac +++ b/configure.ac @@ -582,6 +582,19 @@ fi AC_MSG_RESULT([$DISTRO]) AC_SUBST([DISTRO]) +AC_MSG_CHECKING([whether we're building on Mac OS X]) +if test "x$(uname)" = "xDarwin"; then + AC_MSG_RESULT([yes]) + LCRYPT_FLAG= + SHARED_LIBS_PATH="DYLD_LIBRARY_PATH" +else + AC_MSG_RESULT([no]) + LCRYPT_FLAG="-lcrypt" + SHARED_LIBS_PATH="LD_LIBRARY_PATH" +fi +AC_SUBST([LCRYPT_FLAG]) +AC_SUBST([SHARED_LIBS_PATH]) + dnl Add extra packages to the appliance. AC_ARG_WITH([extra-packages], [AS_HELP_STRING([--with-extra-packages="pkg1 pkg2 ..."], diff --git a/run.in b/run.in index a8c2904..4612e0e 100755 --- a/run.in +++ b/run.in @@ -77,13 +77,13 @@ fi PATH="$b/align:$b/builder:$b/cat:$b/customize:$b/df:$b/diff:$b/edit:$b/erlang:$b/fish:$b/format:$b/fuse:$b/inspector:$b/make-fs:$b/p2v:$b/rescue:$b/resize:$b/sparsify:$b/sysprep:$b/test-tool:$b/tools:$b/v2v:$PATH" export PATH -# Set LD_LIBRARY_PATH to contain library. -if [ -z "$LD_LIBRARY_PATH" ]; then - LD_LIBRARY_PATH="$b/src/.libs:$b/java/.libs:$b/gobject/.libs" +# Set @SHARED_LIBS_PATH@ to contain library. +if [ -z "$@SHARED_LIBS_PATH@" ]; then + @SHARED_LIBS_PATH@="$b/src/.libs:$b/java/.libs:$b/gobject/.libs" else - LD_LIBRARY_PATH="$b/src/.libs:$b/java/.libs:$b/gobject/.libs:$LD_LIBRARY_PATH" + @SHARED_LIBS_PATH@="$b/src/.libs:$b/java/.libs:$b/gobject/.libs:$@SHARED_LIBS_PATH@" fi -export LD_LIBRARY_PATH +export @SHARED_LIBS_PATH@ # Make virt-builder use the local website copy to avoid hitting # the network all the time. diff --git a/v2v/link.sh.in b/v2v/link.sh.in index 79dc847..3971d03 100644 --- a/v2v/link.sh.in +++ b/v2v/link.sh.in @@ -19,4 +19,4 @@ # Hack automake to link binary properly. There is no other way to add # the -cclib parameter to the end of the command line. -exec "$@" -linkpkg -cclib '-lutils @LIBTINFO_LIBS@ -lcrypt @LIBVIRT_LIBS@ @LIBXML2_LIBS@ @LIBINTL@ -lgnu' +exec "$@" -linkpkg -cclib '-lutils @LIBTINFO_LIBS@ @LCRYPT_FLAG@ @LIBVIRT_LIBS@ @LIBXML2_LIBS@ @LIBINTL@ -lgnu' -- 1.9.3

2 3

host Linux - guest Win7 fast file sharing [was: getting guestfs_rsync_out to work]
by Jeff Brower 11 Feb '15

11 Feb '15

Update: Using mount (read-only) and unmount commands in a script that loops continuously, we are able to see near instantaneous file modifications (less than 0.5 sec) performed on a Win7 live guest. We have to use the sync.exe utility (technet.microsoft.com/en-us/sysinternals/bb897438.aspx) otherwise changes are not visible for 10-20 sec. We tested sync.exe with our libguestfs test program (using various methods, including rsync_out, and w/wo sync on Linux side), but we're still not able to see live guest file changes until we relaunch the image. Any advice on how to get around the relaunch requirement (which takes 3-5 sec due to loading the appliance Linux kernel), would be greatly appreciated. There are many reasons why using libguestfs is a better solution for us in the long run. Thanks. -Jeff ---------------- Original Message --------------- Subject: Re: [Libguestfs] getting guestfs_rsync_out to work From: "Jeff Brower" <jbrower(a)signalogic.com> Date: Mon, February 9, 2015 3:39 pm To: "Richard W.M. Jones" <rjones(a)redhat.com> Cc: libguestfs(a)redhat.com ------------------------------------------------- Richard- > On Mon, Feb 09, 2015 at 08:27:22AM -0600, Jeff Brower wrote: >> Yes we know that it doesn't know. We'd be ok to re-launch the image to pick up changes except for the time it takes (if we could get into 1 to 10 msec range it would be ok). > > There's no feasible way to start a Linux kernel in under about 1-2 seconds. It just takes that long for the kernel to initialize itself. Even if you use the User-mode Linux backend (which just runs vmlinux as a userspace process) you'll experience approximately the same overhead. Ok got it. Thanks again for your reply. We tried guestfs_drop_caches() as follows: while(1) { guestfs_mount(g, "/dev/sda2", "/"); printf("%s\n", guestfs_cat(g, "/HostShared/temp.txt")); guestfs_umount(g, "/"); guestfs_drop_caches(g, 3); usleep(1000*1000); } and still we cannot see changes made to temp.txt (from inside the Win7 guest VM) while this loop continues to run on the host Linux. We still have to re-launch the image to see changes. In general, it would be great if the "kernel load" and "disk refresh / sync" functions in guestfs_launch() could be separated. If there was a way to do read-only file sharing, a very limited subset of what virtFs/9p can do, that would be great for Win guests. -Jeff

2 1

[PATCH] php: add a simple bindtests test
by Pino Toscano 11 Feb '15

11 Feb '15

Generate a simple bindtests test for the PHP binding, so it is possible to easily test all the argument types. Unlike the bindtests for other languages, optional arguments are not tested, due to the limitations of optional arguments in PHP (or maybe they way we implement them). --- .gitignore | 1 + generator/bindtests.ml | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++ generator/main.ml | 1 + 3 files changed, 65 insertions(+) diff --git a/.gitignore b/.gitignore index 1b02af0..b165c81 100644 --- a/.gitignore +++ b/.gitignore @@ -387,6 +387,7 @@ Makefile.in /php/extension/php-for-tests.sh /php/extension/php_guestfs_php.h /php/extension/run-tests.php +/php/extension/tests/guestfs_php_bindtests.phpt /php/extension/tmp-php.ini /pick-guests.pl /po-docs/*/*.1 diff --git a/generator/bindtests.ml b/generator/bindtests.ml index b75de15..2e9b6d3 100644 --- a/generator/bindtests.ml +++ b/generator/bindtests.ml @@ -909,6 +909,69 @@ and generate_golang_bindtests () = pr " return &s;\n"; pr "}\n" +and generate_php_bindtests () = + (* No header for this, as it is a .phpt file. *) + + (* Unfortunately, due to the way optional arguments work in PHP, + * we cannot test arbitrary arguments skipping the previous ones + * in the function signatures. + * + * Hence, check only the non-optional arguments, and fix the + * baseline output to expect always "unset" optional arguments. + *) + + pr "--TEST--\n"; + pr "General PHP binding test.\n"; + pr "--FILE--\n"; + pr "<?php\n"; + pr "$g = guestfs_create ();\n"; + + let mkargs args = + String.concat ", " ( + List.map ( + function + | CallString s -> "\"" ^ s ^ "\"" + | CallOptString None -> "NULL" + | CallOptString (Some s) -> sprintf "\"%s\"" s + | CallStringList xs -> + sprintf "array(%s)" + (String.concat "," (List.map (sprintf "\"%s\"") xs)) + | CallInt i -> string_of_int i + | CallInt64 i -> Int64.to_string i + | CallBool b -> if b then "1" else "0" + | CallBuffer s -> "\"" ^ c_quote s ^ "\"" + ) args + ) + in + + generate_lang_bindtests ( + fun f args optargs -> + pr "if (guestfs_%s ($g, %s) == false) {\n" f (mkargs args); + pr " echo (\"Call failed: \" . guestfs_last_error ($g) . \"\\n\");\n"; + pr " exit;\n"; + pr "}\n"; + ); + + pr "echo (\"EOF\\n\");\n"; + pr "?>\n"; + pr "--EXPECT--\n"; + + let dump filename = + let chan = open_in filename in + let rec loop () = + let line = input_line chan in + (match string_split ":" line with + | ("obool"|"oint"|"oint64"|"ostring"|"ostringlist") as x :: _ -> + pr "%s: unset\n" x + | _ -> pr "%s\n" line + ); + loop () + in + (try loop () with End_of_file -> ()); + close_in chan in + + dump "bindtests" + (* Language-independent bindings tests - we do it this way to * ensure there is parity in testing bindings across all languages. *) diff --git a/generator/main.ml b/generator/main.ml index 436d72d..abeae0c 100644 --- a/generator/main.ml +++ b/generator/main.ml @@ -160,6 +160,7 @@ Run it from the top source directory using the command output_to "csharp/Libguestfs.cs" generate_csharp; output_to "php/extension/php_guestfs_php.h" generate_php_h; output_to "php/extension/guestfs_php.c" generate_php_c; + output_to "php/extension/tests/guestfs_php_bindtests.phpt" generate_php_bindtests; output_to "erlang/guestfs.erl" generate_erlang_erl; output_to "erlang/erl-guestfs.c" generate_erlang_c; output_to ~perm:0o555 "erlang/bindtests.erl" generate_erlang_bindtests; -- 1.9.3

2 1

[PATCH 1/4] php: fix invalid memory access with OptString
by Pino Toscano 11 Feb '15

11 Feb '15

OptString maps to a "s!" argument, which makes zend_parse_parameters not touch the variables (char* and length) when NULL is passed as parameter. Hence, set both to NULL/0, and check for non-NULL char* variable before checking its length. --- generator/php.ml | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/generator/php.ml b/generator/php.ml index 0ef6541..1384451 100644 --- a/generator/php.ml +++ b/generator/php.ml @@ -192,11 +192,13 @@ PHP_FUNCTION (guestfs_last_error) | String n | Device n | Mountable n | Pathname n | Dev_or_Path n | Mountable_or_Path n | FileIn n | FileOut n | Key n - | OptString n | BufferIn n | GUID n -> pr " char *%s;\n" n; pr " int %s_size;\n" n + | OptString n -> + pr " char *%s = NULL;\n" n; + pr " int %s_size;\n" n | StringList n | DeviceList n -> pr " zval *z_%s;\n" n; @@ -310,7 +312,7 @@ PHP_FUNCTION (guestfs_last_error) | String n | Device n | Mountable n | Pathname n | Dev_or_Path n | Mountable_or_Path n | FileIn n | FileOut n | Key n - | OptString n | GUID n -> + | GUID n -> (* Just need to check the string doesn't contain any ASCII * NUL characters, which won't be supported by the C API. *) @@ -319,6 +321,15 @@ PHP_FUNCTION (guestfs_last_error) pr " RETURN_FALSE;\n"; pr " }\n"; pr "\n" + | OptString n -> + (* Just need to check the string doesn't contain any ASCII + * NUL characters, which won't be supported by the C API. + *) + pr " if (%s != NULL && strlen (%s) != %s_size) {\n" n n n; + pr " fprintf (stderr, \"libguestfs: %s: parameter '%s' contains embedded ASCII NUL.\\n\");\n" shortname n; + pr " RETURN_FALSE;\n"; + pr " }\n"; + pr "\n" | BufferIn n -> () | StringList n | DeviceList n -> -- 1.9.3

2 4

[PATCH 1/3] generator: add a simple HTML escaping function
by Pino Toscano 11 Feb '15

11 Feb '15

--- generator/utils.ml | 8 +++++++- generator/utils.mli | 3 +++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/generator/utils.ml b/generator/utils.ml index b24ba8c..3a62084 100644 --- a/generator/utils.ml +++ b/generator/utils.ml @@ -360,4 +360,10 @@ let args_of_optargs optargs = | OInt64 n -> Int64 n | OString n -> String n | OStringList n -> StringList n - ) optargs; + ) optargs + +let html_escape text = + let text = replace_str text "&" "&" in + let text = replace_str text "<" "<" in + let text = replace_str text ">" ">" in + text diff --git a/generator/utils.mli b/generator/utils.mli index 5fd40ce..e0f30c3 100644 --- a/generator/utils.mli +++ b/generator/utils.mli @@ -131,3 +131,6 @@ val spaces : int -> string val args_of_optargs : Types.optargs -> Types.args (** Convert a list of optargs into an equivalent list of args *) + +val html_escape : string -> string +(** Escape a text for HTML display. *) -- 1.9.3

2 3

Re: [Libguestfs] getting guestfs_rsync_out to work
by Jeff Brower 10 Feb '15

10 Feb '15

Richard- >>> Per this page >>> >>> https://rwmj.wordpress.com/2013/04/22/using-rsync-with-libguestfs/ >>> >>> We are trying to get the libguestfs API guestfs_rsync_out to work. Weve tried combinations similar to: >>> >>> guestfs_rsync_out(g, /HostShared, rsync://root@10.0.1.90:2999/backup/ -av, -1) >>> >>> surrounding it with APIs as in the recipe examples, for example replacing libguestfs_cat(), with no luck. In the >>> above >>> example the "backup" rsync module name matches the script on Richard's wordpress.com blog page (above link). >>> Typically in the rsync daemon log we see something like: >>> >>> receiving file list >>> sent 17 bytes received 24 bytes total size 0 >>> >>> but no data is transferred. We are successful using rsync from the Linux command line, and we can get libguestfs >>> APIs >>> like guestfs_cat() to work fine. But weve had zero luck with rsync_out over the last week. >> >> I think it's really hard to diagnose problems without turning on >> debugging in libguestfs: >> >> http://libguestfs.org/guestfs-faq.1.html#how-do-i-debug-when-using-the-api >> >>> Unfortunately there are no actual API examples online, no >>> recipes on the libguestfs.org site, etc. Does anyone know >>> where there are some actual libguestfs_rsync_out() working examples? >> >> The only example I can point to is this test: >> >> https://github.com/libguestfs/libguestfs/blob/master/tests/rsync/test-rsync… > > Thanks very much. In our example, the libguestfs API syntax required turns out to be: > > guestfs_rsync_out(g, "/Win7Folder", "rsync://root@10.0.1.90:2999/backup", GUESTFS_RSYNC_OUT_ARCHIVE, 1, -1) > > This produces a trace result: > > libguestfs: trace: rsync_out "/Win7Folder" "rsync://root@10.0.1.90:2999/backup" "archive:true" > > Also, to avoid chgrp and other permission related messages, we had to add: > > uid = 0 > gid = 0 > > in our rsyncd.conf file (we are running rsync daemaon as root). > > Now we are testing guestfs_rsync_out() in a loop that runs every 1 sec. We hope to achieve continuous synchronization > with a live Win7 guest. As I mentioned before, we're only reading from the Win7 guest, and not a lot of data is > involved, maybe a few files up to 50 MByte. These files are not changing size, just contents. [Update] With continuous loop testing, what we found is that we have to shut down and re-launch the image handle to see changes on the Win7 live guest. Unfortunately image re-launch takes time, 3-5 sec (the image size is 50 GByte). I'm assuming this is because libguestfs makes an internal copy of filesystem, and works from that, and doesn't "refresh" this internal copy until re-launching the image handle. Could we create a second partition on the guest that is much smaller, say 50 MByte, attach the image to that, and thus reduce the shutdown and re-launch time into the msec range? Is there another real-time method, not using rsync? As a note, for Linux guest images, we use virtFs, but it seems that WinX doesn't support the 9p filesystem drivers required for virtFs. Thanks. -Jeff

3 7

Patchable build problems on OS X 10.10
by Margaret Lewicka 09 Feb '15

09 Feb '15

Hello, I'm attempting to create a Homebrew formula to get libguestfs to compile on Mac OS X. I've managed to achieve success with several monkey patches, but since Homebrew's policy is to contact maintainers about proper fixes in upstream, I would like to ask if there are any plans to fix these issues. I'm afraid I don't know C well enough to propose decent solutions myself. Here's a list of compilation issues for 1.28.6 on 10.10: * It's not possible to build Java bindings with Java 8, because javadoc fails on GuestFS.java with a hundred or so errors. * ruby/Makefile.in assumes a .so extension for the built file, while on Mac DLEXT appears to default to .bundle. * src/launch-libvirt.c uses SOCK_CLOEXEC and SOCK_NONBLOCK which don't exist on Darwin. * src/launch-libvirt.c needs to #include <sys/un.h> for sockaddr_un. The rest of the issues have been mentioned in https://www.redhat.com/archives/libguestfs/2013-October/msg00042.html, and I've reused RWMJ's patches. * fuse/guestunmount.c uses Linux-specific fusermount and '-v' option for fuser. * src/proto.c -- From Apple's xdr.h: "If your code invokes an xdrproc_t callback, it must be modified to pass a third parameter, which may simply be zero." * OS X does not provide open_memstream(). * src/guestfs-internal-frontend.h and gnulib/lib/error.c need monkey-patching to replace program_name with getprogname(). The patch I'm using right now to get libguestfs to successfully compile is here: https://gist.github.com/shulima/c194c5057e88dbd8f686 Configure flags are --disable-probes, --disable-daemon, --disable-appliance. -- Margaret

4 9

getting guestfs_rsync_out to work
by Jeff Brower 08 Feb '15

08 Feb '15

All- Per this page https://rwmj.wordpress.com/2013/04/22/using-rsync-with-libguestfs/ We are trying to get the libguestfs API guestfs_rsync_out to work. Weve tried combinations similar to: guestfs_rsync_out(g, /HostShared, rsync://root@10.0.1.90:2999/backup/ -av, -1) surrounding it with APIs as in the recipe examples, for example replacing libguestfs_cat(), with no luck. In the above example the "backup" rsync module name matches the script on Richard's wordpress.com blog page (above link). Typically in the rsync daemon log we see something like: receiving file list sent 17 bytes received 24 bytes total size 0 but no data is transferred. We are successful using rsync from the Linux command line, and we can get libguestfs APIs like guestfs_cat() to work fine. But weve had zero luck with rsync_out over the last week. Unfortunately there are no actual API examples online, no recipes on the libguestfs.org site, etc. Does anyone know where there are some actual libguestfs_rsync_out() working examples? Thanks. -Jeff

2 2