March 2014 - Libguestfs - libguestfs List Archives

[PATCH 1/2] builder: move some language-related code into a Languages module
by Pino Toscano 11 Mar '14

11 Mar '14

Mostly code motion, no behaviour changes. --- builder/Makefile.am | 3 +++ builder/languages.ml | 57 +++++++++++++++++++++++++++++++++++++++++++++++++ builder/languages.mli | 21 ++++++++++++++++++ builder/list_entries.ml | 36 ++----------------------------- 4 files changed, 83 insertions(+), 34 deletions(-) create mode 100644 builder/languages.ml create mode 100644 builder/languages.mli diff --git a/builder/Makefile.am b/builder/Makefile.am index 387913c..110b146 100644 --- a/builder/Makefile.am +++ b/builder/Makefile.am @@ -49,6 +49,8 @@ SOURCES = \ index-parser-c.c \ ini_reader.mli \ ini_reader.ml \ + languages.mli \ + languages.ml \ list_entries.mli \ list_entries.ml \ paths.ml \ @@ -101,6 +103,7 @@ deps = \ setlocale.cmx \ ini_reader.cmx \ paths.cmx \ + languages.cmx \ get_kernel.cmx \ downloader.cmx \ sigchecker.cmx \ diff --git a/builder/languages.ml b/builder/languages.ml new file mode 100644 index 0000000..876b23a --- /dev/null +++ b/builder/languages.ml @@ -0,0 +1,57 @@ +(* virt-builder + * Copyright (C) 2013-2014 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + *) + +open Common_utils + +let split_locale loc = + let regex = Str.regexp "^\$[A-Za-z]+\$\$_\\([A-Za-z]+\$\\)?\$\\.\\([A-Za-z0-9-]+\$\\)?\$@\\([A-Za-z]+\$\\)?$" in + let l = ref [] in + if Str.string_match regex loc 0 then ( + let match_or_empty n = + try Str.matched_group n loc with + | Not_found -> "" + in + let lang = Str.matched_group 1 loc in + let territory = match_or_empty 3 in + (match territory with + | "" -> () + | territory -> l := (lang ^ "_" ^ territory) :: !l); + l := lang :: !l; + ); + l := "" :: !l; + List.rev !l + +let languages () = + match Setlocale.setlocale Setlocale.LC_MESSAGES None with + | None -> [""] + | Some locale -> split_locale locale + +let find_notes languages notes = + let notes = List.fold_left ( + fun acc lang -> + let res = List.filter ( + fun (langkey, _) -> + match langkey with + | "C" -> lang = "" + | langkey -> langkey = lang + ) notes in + match res with + | (_, noteskey) :: _ -> noteskey :: acc + | [] -> acc + ) [] languages in + List.rev notes diff --git a/builder/languages.mli b/builder/languages.mli new file mode 100644 index 0000000..f096e9a --- /dev/null +++ b/builder/languages.mli @@ -0,0 +1,21 @@ +(* virt-builder + * Copyright (C) 2014 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + *) + +val languages : unit -> string list + +val find_notes : string list -> (string * string) list -> string list diff --git a/builder/list_entries.ml b/builder/list_entries.ml index 476bf14..623040a 100644 --- a/builder/list_entries.ml +++ b/builder/list_entries.ml @@ -21,24 +21,6 @@ open Common_utils open Printf -let split_locale loc = - let regex = Str.regexp "^\$[A-Za-z]+\$\$_\\([A-Za-z]+\$\\)?\$\\.\\([A-Za-z0-9-]+\$\\)?\$@\\([A-Za-z]+\$\\)?$" in - let l = ref [] in - if Str.string_match regex loc 0 then ( - let match_or_empty n = - try Str.matched_group n loc with - | Not_found -> "" - in - let lang = Str.matched_group 1 loc in - let territory = match_or_empty 3 in - (match territory with - | "" -> () - | territory -> l := (lang ^ "_" ^ territory) :: !l); - l := lang :: !l; - ); - l := "" :: !l; - List.rev !l - let rec list_entries ~list_format ~sources index = match list_format with | `Short -> list_entries_short index @@ -60,9 +42,7 @@ and list_entries_short index = ) index and list_entries_long ~sources index = - let langs = match Setlocale.setlocale Setlocale.LC_MESSAGES None with - | None -> [""] - | Some locale -> split_locale locale in + let langs = Languages.languages () in List.iter ( fun (source, key) -> @@ -97,19 +77,7 @@ and list_entries_long ~sources index = | Some size -> printf "%-24s %s\n" (s_"Download size:") (human_size size); ); - let notes = List.fold_left ( - fun acc lang -> - let res = List.filter ( - fun (langkey, _) -> - match langkey with - | "C" -> lang = "" - | langkey -> langkey = lang - ) notes in - match res with - | (_, noteskey) :: _ -> noteskey :: acc - | [] -> acc - ) [] langs in - let notes = List.rev notes in + let notes = Languages.find_notes langs notes in (match notes with | notes :: _ -> printf "\n"; -- 1.8.3.1

2 3

[PATCH 0/3] Add discard support.
by Richard W.M. Jones 10 Mar '14

10 Mar '14

These patches contain the beginnings of discard (a.k.a. trim or unmap) support. This will allow us to change virt-sparsify to work on disk images in-place (instead of using slow & inefficient copying). The approach used is to add an optional 'discard' parameter to add-drive. It has 3 possible settings: - 'disable' : the default, no discard is done - 'besteffort' : try to enable it, but don't fail if it's not supported - 'enable' : enable it, and fail if it is not supported I'm not sure if we should wait for post-1.26 before adding discard support. It depends if the interface above is suitable or if we'll have to change it later. However I discussed how to support this at DevConf and this approach was generally agreed. Rich.

2 7

virt-builder disk space
by Joshua Daniel Franklin 10 Mar '14

10 Mar '14

Hi, I've got a feature request, it would be nice to have some friendly info in the case you don't have enough disk space on the default locations for virt-builder and supermin to run. I was using virt-builder on the official Fedora AMI on EC2 which includes an awesome 2G root block device and, since RAM is at a premium in the smaller instance types, tmpfs is not used. http://fedoraproject.org/en/get-fedora-options#clouds However, EC2 also includes direct-attached "Instance Storage" scratch space such as 160G magnetic on m1.small or 32G ssd on m3.large (details at http://aws.amazon.com/ec2/pricing/ ) which cloud-init automatically mounts at /mnt. All you get output wise is : virt-builder: libguestfs error: /usr/bin/supermin-helper exited with error status 1. Even with LIBGUESTFS_DEBUG and LIBGUESTFS_TRACE supermin isn't super helpful but you can see lines such as: libguestfs: trace: get_cachedir = "/var/tmp" /usr/bin/supermin-helper: ext2fs_read_inode: Illegal inode number This works fine: export TMPDIR=/mnt virt-builder --cache /mnt fedora-20 More output attached. I love virt-builder btw.

2 1

[PATCH] builder: use Filename.check_suffix
by Pino Toscano 10 Mar '14

10 Mar '14

... instead of manually extracting and checking the suffix by ourselves --- builder/sources.ml | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/builder/sources.ml b/builder/sources.ml index fd0b236..016adc4 100644 --- a/builder/sources.ml +++ b/builder/sources.ml @@ -87,10 +87,7 @@ let merge_sources current_sources new_sources = ) new_sources current_sources let filter_filenames filename = - let suffix = ".conf" in - let n = String.length filename in - let ns = String.length suffix in - n >= ns && String.sub filename (n - ns) ns = suffix + Filename.check_suffix filename ".conf" let read_sources ~prog ~debug = let dirs = Paths.xdg_config_dirs ~prog in -- 1.8.3.1

2 1

FYI: Fwd: [Bug 1157665] Re: update-guestfs-appliance & ligbuestfs-test-tool should be included in libguestfs0
by Richard W.M. Jones 09 Mar '14

09 Mar '14

Hi Hilko, I don't know if you pay attention to or even care about Ubuntu bugs. I subscribe to ones that involve libguestfs, and one here [see below] involves libguestfs packaging in Debian/Ubuntu. Unfortunately Ubuntu LTS has settled on a really old version of libguestfs too ... Rich. ----- Forwarded message from Maarten Fonville <1157665(a)bugs.launchpad.net> ----- Date: Sun, 09 Mar 2014 18:12:17 -0000 From: Maarten Fonville <1157665(a)bugs.launchpad.net> To: rjones(a)redhat.com Subject: [Bug 1157665] Re: update-guestfs-appliance & ligbuestfs-test-tool should be included in libguestfs0 ** Changed in: libguestfs (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are subscribed to libguestfs in Ubuntu. Matching subscriptions: libguestfs bugs in Ubuntu https://bugs.launchpad.net/bugs/1157665 Title: update-guestfs-appliance & ligbuestfs-test-tool should be included in libguestfs0 Status in “libguestfs” package in Ubuntu: Confirmed Bug description: Description: Ubuntu 12.10 Release: 12.10 libguestfs0: 1:1.18.5-2ubuntu1 In Ubuntu 12.10 the appliance creation script (update-guestfs- appliance) and the libguestfs-test-tool are under libguestfs-tools. I'm not sure this is a right choice. I think they should be under libguestfs0 as they are in Debian. All the applications that use the library should work out of the box when you install them using "apt- get install". For example guestmount has libguestfs0 as a dependency but not liguestfs-tools. If you do: apt-get install guestmount you end up installing a program that does not work: $ guestmount -a debian.raw -m /dev/sda1 --ro /mnt libguestfs: error: cannot find any suitable libguestfs supermin, fixed or old-style appliance on LIBGUESTFS_PATH (search path: /usr/lib/guestfs) And the user needs to manually fix this by either installing libguestfs-tools or downloading a binary appliance. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libguestfs/+bug/1157665/+subscrip… ----- End forwarded message ----- -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

1 1

[PATCH] Disable gnulib tests by default
by Richard W.M. Jones 07 Mar '14

07 Mar '14

From: Hilko Bengen <bengen(a)debian.org> RWMJ: - Patch taken from Debian downstream package. - The default in the original patch was to disable the tests. I have changed this to default to enable them instead. - Removed AC_SUBST as the variable was not being used. Cc: Hilko Bengen <bengen(a)debian.org> --- Makefile.am | 5 ++++- configure.ac | 8 ++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/Makefile.am b/Makefile.am index b47a572..020628e 100644 --- a/Makefile.am +++ b/Makefile.am @@ -20,7 +20,10 @@ include $(top_srcdir)/common-rules.mk ACLOCAL_AMFLAGS = -I m4 # Gnulib - must be built and tested before the library. -SUBDIRS = gnulib/lib gnulib/tests +SUBDIRS = gnulib/lib +if ENABLE_GNULIB_TESTS +SUBDIRS += gnulib/tests +endif # Basic source for the library. SUBDIRS += tests/data generator src examples po diff --git a/configure.ac b/configure.ac index afc6718..97c2c70 100644 --- a/configure.ac +++ b/configure.ac @@ -1553,6 +1553,14 @@ dnl For search paths. AC_DEFINE_UNQUOTED([PATH_SEPARATOR],["$PATH_SEPARATOR"], [Character that separates path elements in search paths]) +AC_ARG_ENABLE([gnulib-tests], + [AS_HELP_STRING([--disable-gnulib-tests], + [disable running GNU Portability library tests @<:@default=yes@:>@])], + [ENABLE_GNULIB_TESTS="$enableval"], + [ENABLE_GNULIB_TESTS=yes]) +AM_CONDITIONAL([ENABLE_GNULIB_TESTS],[test "x$ENABLE_GNULIB_TESTS" = "xyes"]) +AC_MSG_RESULT([$ENABLE_GNULIB_TESTS]) + dnl Library versioning. MAX_PROC_NR=`cat $srcdir/src/MAX_PROC_NR` AC_SUBST(MAX_PROC_NR) -- 1.8.5.3

2 2

[PATCH 0/7] java: Various fixes to RStruct/RStructList bindings (RHBZ#1073906).
by Richard W.M. Jones 07 Mar '14

07 Mar '14

See: https://bugzilla.redhat.com/show_bug.cgi?id=1073906 I'm still running the full tests on these. Rich.

1 7

[PATCH] tests: add simple automake to junit xml conversion script
by Pino Toscano 07 Mar '14

07 Mar '14

Add (and ship) a simple ocaml script which can convert the results of automake unit tests (.trs + .log files) to a junit-like .xml file, suitable for importing in CI systems (such as jenkins). --- tests/Makefile.am | 4 +- tests/automake2junit.ml | 128 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+), 1 deletion(-) create mode 100644 tests/automake2junit.ml diff --git a/tests/Makefile.am b/tests/Makefile.am index fd059d5..9603e89 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -17,7 +17,9 @@ # # Written by Richard W.M. Jones <rjones(a)redhat.com> -EXTRA_DIST = $(TESTS) +EXTRA_DIST = \ + automake2junit.ml \ + $(TESTS) TESTS = \ test-basic.sh diff --git a/tests/automake2junit.ml b/tests/automake2junit.ml new file mode 100644 index 0000000..5f32c45 --- /dev/null +++ b/tests/automake2junit.ml @@ -0,0 +1,128 @@ +(* Copyright (C) 2010-2014 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + *) + +open Printf +#load "str.cma" + +let (//) = Filename.concat + +let read_whole_file path = + let buf = Buffer.create 16384 in + let chan = open_in path in + let maxlen = 16384 in + let s = String.create maxlen in + let rec loop () = + let r = input chan s 0 maxlen in + if r > 0 then ( + Buffer.add_substring buf s 0 r; + loop () + ) + in + loop (); + close_in chan; + Buffer.contents buf + +let string_charsplit sep = + Str.split (Str.regexp_string sep) + +let find_trs basedir = + let rec internal_find_trs basedir = + let items = Array.to_list (Sys.readdir basedir) in + let items = List.map (fun x -> basedir // x) items in + let dirs, files = List.partition ( + fun x -> + try Sys.is_directory x + with Sys_error _ -> false + ) items in + let files = List.filter (fun x -> Filename.check_suffix x ".trs") files in + let subdirs_files = List.fold_left ( + fun acc dir -> + (internal_find_trs dir) :: acc + ) [] dirs in + let subdirs_files = List.rev subdirs_files in + List.concat (files :: subdirs_files) + in + internal_find_trs basedir + +let iterate_results trs_files = + let total = ref 0 in + let failures = ref 0 in + let errors = ref 0 in + let skipped = ref 0 in + let buf = Buffer.create 16384 in + List.iter ( + fun file -> + let rec results file = + let content = read_whole_file file in + let lines = string_charsplit "\n" content in + let log = get_log file in + let testname = name_for_test file in + List.iter ( + fun line -> + let line = string_charsplit " " line in + (match line with + | ":test-result:" :: result :: rest -> + let name = String.concat " " rest in + let name = if String.length name > 0 then name else testname in + let print_tag_with_log tag = + Buffer.add_string buf (sprintf " <testcase name=\"%s\">\n" name); + Buffer.add_string buf (sprintf " <%s><![CDATA[%s]]></%s>\n" tag log tag); + Buffer.add_string buf (sprintf " </testcase>\n") + in + (match result with + | "PASS" -> + print_tag_with_log "system-out" + | "SKIP" -> + skipped := !skipped + 1; + print_tag_with_log "skipped" + | "XFAIL" | "FAIL" | "XPASS" -> + failures := !failures + 1; + print_tag_with_log "error" + | "ERROR" | _ -> + errors := !errors + 1; + print_tag_with_log "error" + ); + total := !total + 1 + | _ -> () + ); + ) lines; + and name_for_test filename = + Filename.chop_suffix (Filename.basename filename) ".trs" + and get_log filename = + let log_filename = (Filename.chop_suffix filename ".trs") ^ ".log" in + try read_whole_file log_filename with _ -> "" + in + results file + ) trs_files; + Buffer.contents buf, !total, !failures, !errors, !skipped + +let () = + if Array.length Sys.argv < 3 then ( + printf "%s PROJECTNAME BASEDIR\n" Sys.argv.(0); + exit 1 + ); + let name = Sys.argv.(1) in + let basedir = Sys.argv.(2) in + let trs_files = List.sort compare (find_trs basedir) in + let buf, total, failures, errors, skipped = + iterate_results trs_files in + printf "<?xml version=\"1.0\" encoding=\"UTF-8\"?> +<testsuites> + <testsuite name=\"%s\" tests=\"%d\" failures=\"%d\" skipped=\"%d\" errors=\"%d\"> +%s </testsuite> +</testsuites> +" name total failures skipped errors buf -- 1.8.3.1

2 2

[PATCH] supermin: Handle FTSENT.fts_info correctly -- it is not a bitmask.
by Hilko Bengen 06 Mar '14

06 Mar '14

Found by trying to build a simple appliance using $ supermin --prepare -o supermin.d dash and adding a tarball containing a symbolic link (/init -> /bin/sh). This symbolic link was not transferred to the filesystem. --- src/ext2fs-c.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ext2fs-c.c b/src/ext2fs-c.c index 25f6788..4d1004c 100644 --- a/src/ext2fs-c.c +++ b/src/ext2fs-c.c @@ -233,7 +233,7 @@ supermin_ext2fs_copy_dir_recursively_from_host (value fsv, break; /* Ignore directories being visited in post-order. */ - if (entry->fts_info & FTS_DP) + if (entry->fts_info == FTS_DP) continue; /* Copy the file. */ -- 1.9.0

2 1

Hivex3: Saving values - always string
by Martin Klíma 05 Mar '14

05 Mar '14

Hallo, I working on GUI interface for users to manipulate Windows Registry. For that I choose to use really excellent library hivex3. Just now I'm performing same test to see, if everything is saved correctly. Most of things work really well, but there is problem with saving some values and his types. Description of problem: For saving values is used function : node_set_values or node_set_value This function take 3 arguments, node name, value type and value. It seems that this function accept for value only strings (any other type throw error). The problem lay when I want save values differed then string. For example: value1 = { "key": "TEST_DWORD2(150)", "t": 4, "value": "150" } Result is saving value "150" not like DWORD but like STRING -> 0x313530, which is not valid DWORD value for Win Regedit. This same is with saving binary values, binary or hex is handle like string. So is there any way how to force library to take this type correctly? I'm using: hivex3 - 1.3.7 python - 2.7.5 Thanks for any advice

2 6