October 2014 - Libguestfs - libguestfs List Archives

[PATCH 1/3] Move JSON to mllib
by Pino Toscano 10 Oct '14

10 Oct '14

Move the simple OCaml JSON writer to mllib, so that can be enhanced and used also outside v2v. --- mllib/JSON.ml | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ mllib/JSON.mli | 26 ++++++++++++++++++++++++++ mllib/Makefile.am | 5 ++++- po/POTFILES-ml | 2 +- v2v/JSON.ml | 53 ----------------------------------------------------- v2v/JSON.mli | 26 -------------------------- v2v/Makefile.am | 3 +-- 7 files changed, 85 insertions(+), 83 deletions(-) create mode 100644 mllib/JSON.ml create mode 100644 mllib/JSON.mli delete mode 100644 v2v/JSON.ml delete mode 100644 v2v/JSON.mli diff --git a/mllib/JSON.ml b/mllib/JSON.ml new file mode 100644 index 0000000..5e3a879 --- /dev/null +++ b/mllib/JSON.ml @@ -0,0 +1,53 @@ +(* virt-v2v + * Copyright (C) 2009-2014 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + *) + +(* Poor man's JSON generator. *) + +open Printf + +open Common_utils + +type field = string * json_t +and json_t = String of string | Int of int +and doc = field list + +(* JSON quoting. *) +let json_quote str = + let str = replace_str str "\\" "\\\\" in + let str = replace_str str "\"" "\\\"" in + let str = replace_str str "'" "\\'" in + let str = replace_str str "\008" "\\b" in + let str = replace_str str "\012" "\\f" in + let str = replace_str str "\n" "\\n" in + let str = replace_str str "\r" "\\r" in + let str = replace_str str "\t" "\\t" in + let str = replace_str str "\011" "\\v" in + str + +let string_of_doc fields = + "{ " ^ + String.concat ", " ( + List.map ( + function + | (n, String v) -> + sprintf "\"%s\" : \"%s\"" n (json_quote v) + | (n, Int v) -> + sprintf "\"%s\" : %d" n v + ) fields + ) + ^ " }" diff --git a/mllib/JSON.mli b/mllib/JSON.mli new file mode 100644 index 0000000..1e3a1b3 --- /dev/null +++ b/mllib/JSON.mli @@ -0,0 +1,26 @@ +(* virt-v2v + * Copyright (C) 2009-2014 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + *) + +(** Poor man's JSON generator. *) + +type field = string * json_t +and json_t = String of string | Int of int +and doc = field list + +val string_of_doc : doc -> string + (** Serialize {!doc} object as a string. *) diff --git a/mllib/Makefile.am b/mllib/Makefile.am index ac953ac..653b8aa 100644 --- a/mllib/Makefile.am +++ b/mllib/Makefile.am @@ -32,6 +32,8 @@ SOURCES = \ fsync-c.c \ fsync.mli \ fsync.ml \ + JSON.mli \ + JSON.ml \ mkdtemp.mli \ mkdtemp.ml \ mkdtemp-c.c \ @@ -65,7 +67,8 @@ ocaml_modules = config \ uRI \ mkdtemp \ planner \ - regedit + regedit \ + JSON OBJECTS = \ $(top_builddir)/fish/guestfish-progress.o \ diff --git a/po/POTFILES-ml b/po/POTFILES-ml index df39e7b..d919b6f 100644 --- a/po/POTFILES-ml +++ b/po/POTFILES-ml @@ -25,6 +25,7 @@ customize/perl_edit.ml customize/random_seed.ml customize/timezone.ml customize/urandom.ml +mllib/JSON.ml mllib/common_gettext.ml mllib/common_utils.ml mllib/common_utils_tests.ml @@ -83,7 +84,6 @@ sysprep/sysprep_operation_user_account.ml sysprep/sysprep_operation_utmp.ml sysprep/sysprep_operation_yum_uuid.ml v2v/DOM.ml -v2v/JSON.ml v2v/OVF.ml v2v/cmdline.ml v2v/convert_linux.ml diff --git a/v2v/JSON.ml b/v2v/JSON.ml deleted file mode 100644 index 5e3a879..0000000 --- a/v2v/JSON.ml +++ /dev/null @@ -1,53 +0,0 @@ -(* virt-v2v - * Copyright (C) 2009-2014 Red Hat Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - *) - -(* Poor man's JSON generator. *) - -open Printf - -open Common_utils - -type field = string * json_t -and json_t = String of string | Int of int -and doc = field list - -(* JSON quoting. *) -let json_quote str = - let str = replace_str str "\\" "\\\\" in - let str = replace_str str "\"" "\\\"" in - let str = replace_str str "'" "\\'" in - let str = replace_str str "\008" "\\b" in - let str = replace_str str "\012" "\\f" in - let str = replace_str str "\n" "\\n" in - let str = replace_str str "\r" "\\r" in - let str = replace_str str "\t" "\\t" in - let str = replace_str str "\011" "\\v" in - str - -let string_of_doc fields = - "{ " ^ - String.concat ", " ( - List.map ( - function - | (n, String v) -> - sprintf "\"%s\" : \"%s\"" n (json_quote v) - | (n, Int v) -> - sprintf "\"%s\" : %d" n v - ) fields - ) - ^ " }" diff --git a/v2v/JSON.mli b/v2v/JSON.mli deleted file mode 100644 index 1e3a1b3..0000000 --- a/v2v/JSON.mli +++ /dev/null @@ -1,26 +0,0 @@ -(* virt-v2v - * Copyright (C) 2009-2014 Red Hat Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - *) - -(** Poor man's JSON generator. *) - -type field = string * json_t -and json_t = String of string | Int of int -and doc = field list - -val string_of_doc : doc -> string - (** Serialize {!doc} object as a string. *) diff --git a/v2v/Makefile.am b/v2v/Makefile.am index 8bf8f07..dafe27e 100644 --- a/v2v/Makefile.am +++ b/v2v/Makefile.am @@ -37,7 +37,6 @@ SOURCES_MLI = \ input_libvirt.mli \ input_libvirtxml.mli \ input_ova.mli \ - JSON.mli \ kvmuid.mli \ linux.mli \ modules_list.mli \ @@ -62,7 +61,6 @@ SOURCES_ML = \ xml.ml \ domainxml.ml \ DOM.ml \ - JSON.ml \ kvmuid.ml \ vCenter.ml \ xen.ml \ @@ -122,6 +120,7 @@ BOBJECTS = \ $(top_builddir)/mllib/progress.cmo \ $(top_builddir)/mllib/config.cmo \ $(top_builddir)/mllib/mkdtemp.cmo \ + $(top_builddir)/mllib/JSON.cmo \ $(top_builddir)/customize/urandom.cmo \ $(top_builddir)/customize/random_seed.cmo \ $(top_builddir)/customize/hostname.cmo \ -- 1.9.3

2 4

Re: [Libguestfs] Plan for libguestfs 1.28
by Richard W.M. Jones 10 Oct '14

10 Oct '14

[Please keep replies on the list to help others] On Fri, Oct 10, 2014 at 05:01:22PM +0800, Zhi Yong Wu wrote: > On Fri, Oct 10, 2014 at 4:47 PM, Richard W.M. Jones <rjones(a)redhat.com> wrote: > > On Fri, Oct 10, 2014 at 04:26:20PM +0800, Zhi Yong Wu wrote: > >> You can't make sure that guest never fail or panic forever, right? > > > > Do you mean the guest or the libguestfs appliance? > > > >> That is, you will have to maintain that guest, not customers' guests. > >> This risk is beyond thought. The maintainance cost will be very > >> expensive in our super large scale cloud environment. > >> > >> We are trying not to start a appliance guest when issuing a libguestfs > >> command, such as virt-resize. > > > > There may be some confusion here. Virt-resize doesn't run the guest. > When i issue virt-resize, i exactly saw one guest is started, indisk > and outdisk are attached to this guest.... > > Do you mean that in the latest upstream, virt-resize doesn't run the guest? OK so you mean the libguestfs appliance. > > TBH I'm not exactly sure what your concerns are. > > We are afraid that if a guest is started when vrit-xxx command is > issued, it will bring some unexpected risks and take too much time to > complete this command. If you have any concrete problems, then let us know. You may wish to limit the number of virt-resize processes than can run at any one time. GNU 'sem' can do this relatively easily: https://www.gnu.org/software/parallel/sem.html There are various tips on how to manage the appliance discussed in this manual page: http://libguestfs.org/guestfs-performance.1.html Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

1 0

Plan for libguestfs 1.28
by Richard W.M. Jones 10 Oct '14

10 Oct '14

It has been an amazing 6½ months since the last stable release of libguestfs. I'd like to plan a new 1.28 release soon. Please follow-up if there are features / blockers / bugs that need to be addressed for 1.28. As usual, bugs which have "1.28" (without quotes) in the Devel Whiteboard field in Bugzilla are nominated as blockers for the release. You can see a list of those here [currently empty]: https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&… or http://tinyurl.com/llyagun For a list of all bugs, go to: https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtua… Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

5 20

KVM Forum
by Richard W.M. Jones 10 Oct '14

10 Oct '14

I'm in Dusseldorf for the KVM Forum (actually, a few days before, starting tomorrow). See you there! http://events.linuxfoundation.org/events/kvm-forum Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

1 0

Re: [Libguestfs] Virt-v2v conversion issue
by Richard W.M. Jones 09 Oct '14

09 Oct '14

[Please keep replies on the list] On Thu, Oct 09, 2014 at 09:51:54PM +0000, VONDRA Alain wrote: > I still have an issue during the update even after a yum clean all : > > Erreur : Paquet : libvirt-python-1.1.1-29.el7_0.3.x86_64 (@updates) > Requiert : libvirt.so.0(LIBVIRT_PRIVATE_1.1.1)(64bit) > Suppression : libvirt-client-1.1.1-29.el7_0.3.x86_64 (@updates) > libvirt.so.0(LIBVIRT_PRIVATE_1.1.1)(64bit) > Mis à jour par : libvirt-client-1.2.8-4.el7.x86_64 (libguestfs-RHEL-7.1-preview) > ~libvirt.so.0(LIBVIRT_PRIVATE_1.2.8)(64bit) > Disponible : libvirt-client-1.1.1-29.el7.x86_64 (base) > libvirt.so.0(LIBVIRT_PRIVATE_1.1.1)(64bit) > Disponible : libvirt-client-1.1.1-29.el7_0.1.x86_64 (updates) > libvirt.so.0(LIBVIRT_PRIVATE_1.1.1)(64bit) > Vous pouvez essayer d'utiliser --skip-broken pour contourner le problème > Vous pouvez essayer d'exécuter : rpm -Va --nofiles --nodigest Apparently now libvirt-python is a separate package. I'll upload libvirt-python at some point soon. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

1 1

[PATCH] log: tests: skip fedora.img if journal is not available
by Pino Toscano 06 Oct '14

06 Oct '14

--- cat/test-virt-log.sh | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/cat/test-virt-log.sh b/cat/test-virt-log.sh index af6bb95..d4fa0ed 100755 --- a/cat/test-virt-log.sh +++ b/cat/test-virt-log.sh @@ -19,7 +19,32 @@ export LANG=C set -e +can_handle () +{ + fn=$(basename $1) + case "$fn" in + fedora.img) + guestfish -a /dev/null run : available journal + ;; + *) + return 0 + ;; + esac +} + # Read out the log files from the image using virt-log. for f in ../tests/guests/{fedora,debian,ubuntu}.img; do - if [ -s "$f" ]; then $VG virt-log -a "$f"; fi + echo "Trying $f ..." + if [ ! -s "$f" ]; then + echo "SKIP: empty file" + echo + continue + fi + if ! can_handle "$f"; then + echo "SKIP: cannot handle $f" + echo + continue + fi + $VG virt-log -a "$f" + echo done -- 1.9.3

2 1

[PATCH] tools: fix free -m invocation
by Pino Toscano 06 Oct '14

06 Oct '14

Since procps 3.3.10, free does not output the "-/+ buffers/cache" line anymore. On the other hand, the data from it can be calculated from the memory values, so just do the calculation manually (with awk). --- df/estimate-max-threads.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/df/estimate-max-threads.c b/df/estimate-max-threads.c index bbcdd73..7d5b090 100644 --- a/df/estimate-max-threads.c +++ b/df/estimate-max-threads.c @@ -44,7 +44,7 @@ estimate_max_threads (void) /* Choose the number of threads based on the amount of free memory. */ mbytes_str = read_line_from ("LANG=C free -m | " - "grep 'buffers/cache' | awk '{print $NF}'"); + "grep '^Mem' | awk '{print $4+$6+$7}'"); if (mbytes_str == NULL) return 1; -- 1.9.3

2 1

[PATCH v4 0/4] tests: Introduce test harness for running tests.
by Richard W.M. Jones 04 Oct '14

04 Oct '14

This converts more of the tests (basically everything under tests/) to use the test harness, revealing some problems which have subsequently been fixed. Rich.

1 4

[PATCH v3] tests: Introduce test harness for running tests.
by Richard W.M. Jones 03 Oct '14

03 Oct '14

We would like to have a more flexible way to run tests, including running them on an installed copy of libguestfs, running them in parallel, and being able to express dependencies and ordering between tests and data files properly. Therefore introduce a test harness (test-harness) program which can run tests either from the locally built copy, or from an installed copy of the tests (in $libdir/guestfs/tests). The test harness is backwards compatible on the command line, ie. 'make check', 'make -C inspector check-valgrind' etc. will still work. But in addition, you can now run the tests on an installed copy of libguestfs by doing: cd $libdir/guestfs/tests ./test-harness The test-harness script supports various options, see 'test-harness --help' for full details. Other notable features: - Checking SKIP_* environment variables in tests is no longer necessary. The test harness deals with these. - Every test runs in its own temporary directory. There is no need to clean up output files. On the other hand, tests must use $srcdir to refer to test data. This is only implemented for a single directory at the moment (ie. inspector/) --- .gitignore | 5 +- Makefile.am | 1 + common-rules.mk | 3 + configure.ac | 2 + generator/Makefile.am | 24 + generator/main.ml | 9 + generator/test_harness.ml | 609 +++++++++++++++++++++++ generator/tests.ml | 64 +++ generator/tests_mk.ml | 104 ++++ generator/types.ml | 9 + inspector/Makefile.am | 17 +- inspector/test-virt-inspector-local-guests.sh.in | 26 + inspector/test-virt-inspector.sh | 13 +- inspector/test-xmllint.sh.in | 5 + inspector/tests.mk | 83 +++ tests/data/Makefile.am | 10 +- tests/guests/Makefile.am | 84 +--- tests/guests/guest-aux/make-debian-img.sh | 6 +- tests/guests/guest-aux/make-fedora-img.pl | 21 +- tests/guests/guest-aux/make-ubuntu-img.sh | 4 +- tests/guests/guest-aux/make-windows-img.sh | 6 +- 21 files changed, 998 insertions(+), 107 deletions(-) create mode 100644 generator/test_harness.ml create mode 100644 generator/tests.ml create mode 100644 generator/tests_mk.ml create mode 100755 inspector/test-virt-inspector-local-guests.sh.in create mode 100644 inspector/tests.mk diff --git a/.gitignore b/.gitignore index 269d735..e4b8881 100644 --- a/.gitignore +++ b/.gitignore @@ -263,8 +263,8 @@ Makefile.in /html/virt-tar-out.1.html /html/virt-v2v.1.html /html/virt-win-reg.1.html -/inspector/actual-*.xml /inspector/stamp-virt-inspector.pod +/inspector/test-virt-inspector-local-guests.sh /inspector/test-xmllint.sh /inspector/virt-inspector /inspector/virt-inspector.1 @@ -475,6 +475,7 @@ Makefile.in /sysprep/virt-sysprep.1 /test.err /test.out +/test-harness /tests/c-api/test-add-drive-opts /tests/c-api/test-add-libvirt-dom /tests/c-api/test-backend-settings @@ -524,7 +525,7 @@ Makefile.in /tests/guests/guest-aux/fedora-packages.db /tests/guests/guest-aux/windows-software /tests/guests/guest-aux/windows-system -/tests/guests/stamp-fedora-md.img +/tests/guests/stamp-guests /tests/guests/ubuntu.img /tests/guests/windows.img /tests/mount-local/test-parallel-mount-local diff --git a/Makefile.am b/Makefile.am index d55d8d6..0d6a1de 100644 --- a/Makefile.am +++ b/Makefile.am @@ -201,6 +201,7 @@ EXTRA_DIST = \ logo/fish-5yrs.svg logo/fish-5yrs.png \ logo/virt-builder.svg \ m4/.gitignore \ + test-harness \ tests/run-xml-to-junit.sh \ tests/run-xml-to-junit.xsl \ tmp/.gitignore \ diff --git a/common-rules.mk b/common-rules.mk index 312107e..5a239ab 100644 --- a/common-rules.mk +++ b/common-rules.mk @@ -27,3 +27,6 @@ builddir ?= @builddir@ abs_builddir ?= @abs_builddir@ srcdir ?= @srcdir@ abs_srcdir ?= @abs_srcdir@ + +# Tests directory. +alltestsdir = $(libdir)/guestfs/tests diff --git a/configure.ac b/configure.ac index 0d99aa0..c717981 100644 --- a/configure.ac +++ b/configure.ac @@ -1618,6 +1618,8 @@ mkdir -p \ dnl http://www.mail-archive.com/automake@gnu.org/msg10204.html AC_CONFIG_FILES([appliance/libguestfs-make-fixed-appliance], [chmod +x,-w appliance/libguestfs-make-fixed-appliance]) +AC_CONFIG_FILES([inspector/test-virt-inspector-local-guests.sh], + [chmod +x,-w inspector/test-virt-inspector-local-guests.sh]) AC_CONFIG_FILES([inspector/test-xmllint.sh], [chmod +x,-w inspector/test-xmllint.sh]) AC_CONFIG_FILES([p2v/virt-p2v-make-disk], diff --git a/generator/Makefile.am b/generator/Makefile.am index 3716c77..3bcff61 100644 --- a/generator/Makefile.am +++ b/generator/Makefile.am @@ -52,6 +52,9 @@ sources = \ ruby.ml \ structs.ml \ structs.mli \ + tests.ml \ + tests_mk.ml \ + test_harness.ml \ tests_c_api.ml \ types.ml \ utils.ml \ @@ -71,6 +74,8 @@ objects = \ pr.cmo \ docstrings.cmo \ checks.cmo \ + tests.cmo \ + tests_mk.cmo \ c.cmo \ xdr.cmo \ daemon.cmo \ @@ -93,17 +98,30 @@ objects = \ customize.cmo \ main.cmo +test_harness_objects = \ + types.cmo \ + utils.cmo \ + tests.cmo \ + test_harness.cmo + EXTRA_DIST = $(sources) files-generated.txt OCAMLCFLAGS = -warn-error CDEFLMPSUVYZX-3 -I $(srcdir) -I . -package unix,str noinst_PROGRAM = generator +# Install the test harness. +localtestsdir = $(alltestsdir) +localtests_SCRIPTS = ../test-harness + if HAVE_OCAML generator: $(objects) $(OCAMLFIND) ocamlc $(OCAMLCFLAGS) -linkpkg $^ -o $@ +../test-harness: $(test_harness_objects) + $(OCAMLFIND) ocamlc $(OCAMLCFLAGS) -linkpkg $^ -o $@ + # Dependencies. %.cmi: %.mli $(OCAMLFIND) ocamlc $(OCAMLCFLAGS) -c $< -o $@ @@ -136,6 +154,12 @@ generator: chmod +x $@-t mv $@-t $@ +../test-harness: + rm -f $@ $@-t + echo 'echo Warning: Install OCaml compiler in order to rebuild the test-harness.' > $@-t + chmod +x $@-t + mv $@-t $@ + endif noinst_DATA = stamp-generator diff --git a/generator/main.ml b/generator/main.ml index c0ad146..3ae2ff7 100644 --- a/generator/main.ml +++ b/generator/main.ml @@ -30,6 +30,8 @@ open Types open C open Xdr open Daemon +open Tests +open Tests_mk open Tests_c_api open Fish open Ocaml @@ -48,6 +50,8 @@ open Bindtests open Errnostring open Customize +let (//) = Filename.concat + let perror msg = function | Unix_error (err, _, _) -> eprintf "%s: %s\n" msg (error_message err) @@ -214,6 +218,11 @@ Run it from the top source directory using the command output_to "customize/customize-synopsis.pod" generate_customize_synopsis_pod; output_to "customize/customize-options.pod" generate_customize_options_pod; + List.iter ( + fun (dir, tests) -> + output_to (dir // "tests.mk") (generate_tests_mk dir tests) + ) tests; + (* Generate the list of files generated -- last. *) printf "generated %d lines of code\n" (get_lines_generated ()); let files = List.sort compare (get_files_generated ()) in diff --git a/generator/test_harness.ml b/generator/test_harness.ml new file mode 100644 index 0000000..4786ff1 --- /dev/null +++ b/generator/test_harness.ml @@ -0,0 +1,609 @@ +(* libguestfs + * Copyright (C) 2014 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + *) + +(* The test-harness standalone program. *) + +open Unix +open Printf + +open Types +open Tests + +let (//) = Filename.concat + +let is_dir path = + try (stat path).st_kind = S_DIR + with Unix_error _ -> false + +let is_file path = + try (stat path).st_kind = S_REG + with Unix_error _ -> false + +let is_executable path = + try (stat path).st_perm land 0o111 <> 0 + with Unix_error _ -> false + +let relative_path_to_absolute path = + let cmd = sprintf "unset CDPATH; cd %s && pwd" (Filename.quote path) in + let chan = open_process_in cmd in + let path = input_line chan in + (match close_process_in chan with + | WEXITED 0 -> () + | WEXITED _ + | WSIGNALED _ + | WSTOPPED _ -> + failwith (sprintf "failed to convert relative path to absolute path: %s" + cmd) + ); + path + +let mkdtemp () = + let chan = open_process_in "mktemp -d" in + let path = input_line chan in + (match close_process_in chan with + | WEXITED 0 -> () + | WEXITED _ + | WSIGNALED _ + | WSTOPPED _ -> + failwith "mktemp -d command failed" + ); + path + +type ('a, 'b) maybe = Either of 'a | Or of 'b + +let pushdir dir f = + let olddir = getcwd () in + chdir dir; + let r = try Either (f ()) with exn -> Or exn in + chdir olddir; + match r with + | Either r -> r + | Or exn -> raise exn + +(* Timeout settings. *) +let timeout_period = "4h" +and timeout_kill = "30s" + +type start_dir = +| TopDir (* top dir - run all the tests *) +| PhonyGuestsDir (* phony guests dir *) +| Dir of string * test (* a test directory *) + +let () = + let home = + try Sys.getenv "HOME" + with Not_found -> + failwith "HOME environment variable is not set" in + + let debug = ref false in + let direct = ref false in + let fast = ref false in + let local_guests = ref false in + let make_phony_guests_only = ref false in + let slow = ref false in + let uml = ref false in + let upstream_libvirt = ref false in + let upstream_qemu = ref false in + let valgrind = ref false in + let uml_binary = ref (home // "d/linux-um/vmlinux") in + let libvirtdir = ref (home // "d/libvirt") in + let qemu_binary = ref (home // "d/qemu/x86_64-softmmu/qemu-system-x86_64") in + + let argspec = Arg.align [ + "--debug", Arg.Set debug, " Run tests with debugging enabled"; + "--direct", Arg.Set direct, " Run tests using the direct backend"; + "--fast", Arg.Set fast, " Run only tests which do not need the appliance"; + "--local-guests", Arg.Set local_guests, " Run tests that use locally installed guests r/o"; + "--make-phony-guests-only", Arg.Set make_phony_guests_only, " Generate the phony guests used for testing"; + "--slow", Arg.Set slow, " Run only long-running tests"; + "--uml", Arg.Set uml, " Run tests using UML backend"; + "--upstream-libvirt", Arg.Set upstream_libvirt, " Run tests using upstream libvirt"; + "--upstream-qemu", Arg.Set upstream_qemu, " Run tests using upstream qemu"; + "--valgrind", Arg.Set valgrind, " Run tests under valgrind"; + "--with-uml", Arg.Set_string uml_binary, "vmlinux Select UML binary"; + "--with-upstream-libvirt", Arg.Set_string libvirtdir, "dir Select libvirt directory"; + "--with-upstream-qemu", Arg.Set_string qemu_binary, "qemu Select qemu binary"; + ] in + let args = ref [] in + let anon_fun s = args := s :: !args in + let usage_msg = + sprintf "\ +test-harness: Run the libguestfs test suite, or parts of it. + +Usage: + + test-harness [--options] [directory] + +Normally tests in the current directory are run. If the current +directory is the top level of the test suite then the whole test +suite is run. If the current directory is a leaf directory, then +only tests in that directory are run. + +If you specify a directory name on the command line, then we run +tests from that directory instead. + +Options to select which tests to run: + + --fast Run only tests which do not need the appliance + --slow Run only long-running tests +(if neither --fast nor --slow, then --fast + normal tests are run) + --local-guests Run tests that use locally installed guests r/o + +Options to run the selected tests under different test conditions: + + --debug Run tests with debugging enabled [recommended] + --valgrind Run tests under valgrind + --direct Run tests using the direct backend + --uml Run tests using the UML backend + --with-uml VMLINUX Select UML binary + [default %s] + --upstream-libvirt Run tests using upstream libvirt + --with-upstream-libvirt LIBVIRTDIR + Select libvirt directory + [default %s] + --upstream-qemu Run tests using upstream qemu + --with-upstream-qemu QEMUBINARY + Select qemu binary + [default %s] + +Options used internally: + + --make-phony-guests-only + Generate the phony guests used for testing + +Full list of options in alphabetical order:" + !uml_binary !libvirtdir !qemu_binary in + Arg.parse argspec anon_fun usage_msg; + + let args = !args in + let debug = !debug in + let direct = !direct in + let fast = !fast in + let local_guests = !local_guests in + let make_phony_guests_only = !make_phony_guests_only in + let slow = !slow in + let uml = !uml in + let upstream_libvirt = !upstream_libvirt in + let upstream_qemu = !upstream_qemu in + let valgrind = !valgrind in + let uml_binary = !uml_binary in + let libvirtdir = !libvirtdir in + let qemu_binary = !qemu_binary in + + (* Some combinations are not permitted. *) + if fast && slow || fast && local_guests || slow && local_guests then + failwith "cannot use --fast, --slow and --local-guests options together"; + + (* If none of the selection options are used, default to fast + normal. *) + let fast, normal = + if not fast && not slow && not local_guests then true, true + else fast, false in + + (* --direct and --uml cannot be combined. *) + if direct && uml then + failwith "cannot use --direct and --uml options together"; + + (* If there is a single parameter on the command line, then we + * chdir to that directory. + *) + (match args with + | [] -> () + | [dir] -> chdir dir + | _ -> + failwith "too many command line arguments" + ); + + (* Which directory are we running in? *) + let start_dir = + let pwd = getcwd () in + let basename = Filename.basename pwd in + if is_executable "test-harness" && is_dir "inspector" then + TopDir + else if basename = "guests" && is_dir "guest-aux" then + PhonyGuestsDir + else ( + (* Search through the tests for the current directory. *) + try + let dir, tests = + List.find ( + fun (dir, tests) -> + (* Find the name of any test that should be in the directory. *) + let any_file = + if tests.check <> [] then List.hd tests.check + else if tests.check_fast <> [] then List.hd tests.check_fast + else if tests.check_slow <> [] then List.hd tests.check_slow + else if tests.check_local_guests <> [] then + List.hd tests.check_local_guests + else ( + assert (tests.check_data <> []); + List.hd tests.check_data + ) in + basename = dir && is_file any_file + ) tests in + Dir (dir, tests) + with + Not_found -> + failwith (sprintf "current directory (%s) is not a libguestfs test directory" basename) + ) in + + (* If we are running from automake, then automake will pass $srcdir + * to us, and if it's not "." then we have to adjust our path to the + * top source directory accordingly. + *) + let srcdir = try Sys.getenv "srcdir" with Not_found -> "." in + + (* Are we running from the build directory or from installed tests? *) + let running_in_builddir = is_file (srcdir // "Makefile.am") in + + (* If installed, then we cannot write to the phony guests directory. *) + let phonydir = + if running_in_builddir then ( + match start_dir with + | TopDir -> + relative_path_to_absolute "tests/guests" + | PhonyGuestsDir -> + relative_path_to_absolute "." + | Dir (dir, _) -> + let top_builddir = ref ".." in + for i = 0 to String.length dir-1 do + if dir.[i] = '/' then top_builddir := !top_builddir // ".." + done; + let top_builddir = !top_builddir in + relative_path_to_absolute (top_builddir // "tests/guests") + ) + else ( + (try + mkdir (home // ".cache") 0o755; + mkdir (home // ".cache/libguestfs-tests") 0o755; + mkdir (home // ".cache/libguestfs-tests/phony-guests") 0o755; + with Unix_error _ -> () + ); + home // ".cache/libguestfs-tests/phony-guests" + ) in + + (* If the user gave the valgrind option, check it is valid. *) + if valgrind then ( + if Sys.command "valgrind --help >/dev/null 2>&1" <> 0 then + failwith "valgrind is not installed" + ); + + (* Do we have Padraig's timeout utility? It must have the + * --foreground option (RHBZ#1025269) and the -k option (not present + * in RHEL 6). + *) + let timeout = + Sys.command "timeout --help >/dev/null 2>&1" = 0 && + Sys.command "timeout --foreground 2 sleep 0 >/dev/null 2>&1" = 0 && + Sys.command "timeout -k 10s 10s true >/dev/null 2>&1" = 0 in + + (* Set up the environment (variables) for a test. *) + let set_up_environment () = + let abs_builddir = relative_path_to_absolute "." in + let abs_srcdir = relative_path_to_absolute srcdir in + + (* Work out how to get to the top source directory from the current + * directory. + *) + let rec loop i top_builddir = + if i > 10 then + failwith "set_up_environment: not in a test directory" + else if is_executable (top_builddir // "test-harness") then + top_builddir + else if top_builddir = "." then + loop (i+1) ".." + else + loop (i+1) (top_builddir // "..") + in + let top_builddir = loop 0 "." in + let top_srcdir = + if srcdir = "." then top_builddir + else top_builddir // srcdir in + let abs_top_builddir = relative_path_to_absolute top_builddir in + let abs_top_srcdir = relative_path_to_absolute top_srcdir in + + let datadir = abs_top_builddir // "tests/data" in + + (* Note that because the tests always run in a temporary + * directory, relative paths are useless, so we always populate the + * environment with absolute paths. + *) + putenv "builddir" abs_builddir; + putenv "srcdir" abs_srcdir; + putenv "top_builddir" abs_top_builddir; + putenv "top_srcdir" abs_top_srcdir; + putenv "abs_builddir" abs_builddir; + putenv "abs_srcdir" abs_srcdir; + putenv "abs_top_builddir" abs_top_builddir; + putenv "abs_top_srcdir" abs_top_srcdir; + putenv "phonydir" phonydir; + putenv "datadir" datadir; + + (* Debugging? *) + if debug then ( + putenv "LIBGUESTFS_DEBUG" "1"; + putenv "LIBGUESTFS_TRACE" "1" + ); + + (* Direct backend? *) + if direct then + putenv "LIBGUESTFS_BACKEND" "direct" + + (* UML? *) + else if uml then ( + putenv "LIBGUESTFS_BACKEND" "uml"; + putenv "LIBGUESTFS_HV" uml_binary + ); + + (* Upstream QEMU? *) + if upstream_qemu then + putenv "LIBGUESTFS_HV" qemu_binary; + in + + (* Function which runs to create the phony guests for testing. + * Unfortunately we're recreating 'make' here. This isn't really + * avoidable as we have to be able to run after installation, and + * it's inconvenient to create a Makefile for that. + *) + let rec make_phony_guests () = + (* Set up the environment as for tests ... *) + set_up_environment (); + (* ... but adjust srcdir to point to the tests/guests directory + * since the scripts in guest-aux are expecting this. + *) + let abs_top_srcdir = Sys.getenv "abs_top_srcdir" in + putenv "srcdir" (abs_top_srcdir // "tests/guests"); + + pushdir phonydir (fun () -> + (* Because we distribute all the intermediate files, they are + * always(?) in srcdir, not builddir. + *) + let gaux = abs_top_srcdir // "tests/guests/guest-aux" in + + (* Make several different blank images. These are not guests, but we + * include them in the libvirt fake XML to make sure that virt-df and + * virt-alignment-scan don't break when they encounter them. + *) + List.iter ( + fun ft -> + let o = sprintf "blank-%s.img" ft in + rule o [] (fun () -> + let cmd = + sprintf "guestfish -N %s-t=%s exit && mv %s-t %s" o ft o o in + if Sys.command cmd <> 0 then + failwith "guestfish command failed" + ) + ) [ "disk"; "part"; "fs"; "bootroot"; "bootrootlv" ]; + + (* Make several (phony) Fedora images. *) + List.iter ( + fun (layout, o) -> + rule o [ gaux // "make-fedora-img.pl"; + gaux // "fedora-journal.tar.xz"; + gaux // "fedora-name.db"; + gaux // "fedora-packages.db" ] + (fun () -> + let cmd = sprintf "%s/make-fedora-img.pl --layout=%s" + gaux layout in + if Sys.command cmd <> 0 then + failwith "make-fedora-img.pl failed" + ) + ) [ "partitions", "fedora.img"; + "partitions-md", "fedora-md1.img"; + "btrfs", "fedora-btrfs.img" ]; + + (* Make a (phony) Debian image. *) + let o = "debian.img" in + rule o [ gaux // "make-debian-img.sh" ] + (fun () -> + let cmd = sprintf "%s/make-debian-img.sh" gaux in + if Sys.command cmd <> 0 then + failwith "make-debian-img.sh failed" + ); + + (* Make a (phony) Ubuntu image. *) + let o = "ubuntu.img" in + rule o [ gaux // "make-ubuntu-img.sh" ] + (fun () -> + let cmd = sprintf "%s/make-ubuntu-img.sh" gaux in + if Sys.command cmd <> 0 then + failwith "make-ubuntu-img.sh failed" + ); + + (* Make a (phony) Windows image. *) + let o = "windows.img" in + rule o [ gaux // "make-windows-img.sh"; + gaux // "windows-software"; + gaux // "windows-system" ] + (fun () -> + let cmd = sprintf "%s/make-windows-img.sh" gaux in + if Sys.command cmd <> 0 then + failwith "make-windows-img.sh failed" + ); + + (* Make XML describing all guests we managed to create above. *) + let cmd = + sprintf "%s/make-guests-all-good.pl > guests-all-good.xml blank-disk.img blank-part.img blank-fs.img blank-bootroot.img blank-bootrootlv.img debian.img fedora.img fedora-md1.img fedora-md2.img fedora-btrfs.img ubuntu.img windows.img" + gaux in + if Sys.command cmd <> 0 then + failwith "make-guests-all-good.pl failed"; + + let cmd = "rm -f *.tmp.*" in + ignore (Sys.command cmd); + ) + + (* Like 'make', run a rule if target does not exist, or if target is + * older than any of the sources. + *) + and rule target sources f = + let target_stat = + try Some (stat target) + with Unix_error (ENOENT, _, _) -> + (* Target file does not exist. *) + printf "test-harness: building %s\n%!" target; + f (); None in + match target_stat with + | None -> () + | Some target_stat -> + let rec loop = function + | [] -> () + | source :: sources -> + let source_stat = stat source in + if target_stat.st_mtime < source_stat.st_mtime then ( + printf "test-harness: building %s\n%!" target; + f () + ) + else + loop sources + in + loop sources + in + + let null_results = (0, 0, 0, 0, 0) in + let add_results (t, s, f, tdo, sk) (t', s', f', tdo', sk') = + (t+t', s+s', f+f', tdo+tdo', sk+sk') + in + + let rec run_all_tests () = + List.fold_left ( + fun results (dir, tests) -> + printf "test-harness: entering directory %s\n%!" dir; + let results = + pushdir dir (fun () -> + add_results results (run_dir_tests dir tests) + ) in + printf "test-harness: leaving directory %s\n%!" dir; + results + ) null_results tests + + (* Run all of the tests in a single directory. 'dir' is the + * directory name, and we are chdir'd into this directory + * already. + *) + and run_dir_tests dir tests = + let results = null_results in + let accumulate = + List.fold_left (fun results t -> add_results results (run_one_test dir t)) + in + let results = + if fast then accumulate results tests.check_fast + else results in + let results = + if normal then accumulate results tests.check + else results in + let results = + if slow then accumulate results tests.check_slow + else results in + let results = + if local_guests then accumulate results tests.check_local_guests + else results in + results + + (* Run a single test. *) + and run_one_test dir test = + let skip_env = try Sys.getenv (skip_name test) with Not_found -> "" in + if skip_env = "1" then ( + printf "SKIP: %s\n%!" test; + (1, 0, 0, 0, 1) + ) + else ( + set_up_environment (); + + (* We run each test in its own temporary directory. *) + let tmpdir = mkdtemp () in + + let results, clean_up_tmpdir = + pushdir tmpdir (fun () -> + let cmd = + sprintf "%s%s$abs_srcdir/%s > output 2>&1" + (if upstream_libvirt then libvirtdir // "run " else "") + (if timeout then + sprintf "timeout --foreground -k %s %s " + timeout_kill timeout_period + else "") + test in + + let start_t = gettimeofday () in + let r = Sys.command cmd in + let end_t = gettimeofday () in + + let secs = end_t -. start_t in + + match r with + | 0 -> (* successful *) + printf "PASS: %s (%.1f seconds)\n%!" test secs; + (1, 1, 0, 0, 0), true + | 77 -> (* skipped *) + ignore (Sys.command "cat output"); + printf "SKIP: %s\n%!" test; + (1, 0, 0, 0, 1), true + | 124 -> (* timed out *) + ignore (Sys.command "cat output"); + eprintf "command timed out after %s\n" timeout_period; + eprintf "TIMED OUT: %s\n%!" test; + (1, 0, 0, 1, 0), true + | r -> (* error *) + ignore (Sys.command "cat output"); + eprintf "command failed with exit code %d\n" r; + eprintf "test results left in %s\n" tmpdir; + eprintf "FAIL: %s\n%!" test; + (1, 0, 1, 0, 0), false + ) in + + if clean_up_tmpdir then + ignore (Sys.command (sprintf "rm -rf %s" (Filename.quote tmpdir))); + + results + ) + + (* Convert a test name like 'test-network.sh' into 'SKIP_TEST_NETWORK_SH'. *) + and skip_name name = + let skip = Utils.replace_char name '-' '_' in + let skip = Utils.replace_char skip '.' '_' in + let skip = "SKIP_" ^ String.uppercase skip in + skip + in + + (* Make the phony guests? *) + if make_phony_guests_only || normal || slow then ( + make_phony_guests (); + if make_phony_guests_only then exit 0 + ); + + (* Run the tests. *) + let total, successful, failed, timedout, skipped = + match start_dir with + | TopDir -> run_all_tests () + | Dir (dir, tests) -> run_dir_tests dir tests + | PhonyGuestsDir -> (0, 0, 0, 0, 0) (* do nothing in this directory *) in + + (* Print the test results. *) + printf "--------------------------------------------------\n"; + printf " TEST SUMMARY\n"; + printf "--------------------------------------------------\n"; + printf "Total tests run . . . . . . . %d\n" total; + printf "Successful . . . . . . . . . . %d\n" successful; + if failed > 0 then + printf "Errors . . . . . . . . . . . . %d\n" failed; + if timedout > 0 then + printf "Timed out . . . . . . . . . . %d\n" timedout; + if skipped > 0 then + printf "Skipped . . . . . . . . . . . %d\n" skipped; + printf "--------------------------------------------------\n"; + + (* If there were any errors, then exit with a failure. *) + exit (if failed = 0 && timedout = 0 then 0 else 1) diff --git a/generator/tests.ml b/generator/tests.ml new file mode 100644 index 0000000..42731b0 --- /dev/null +++ b/generator/tests.ml @@ -0,0 +1,64 @@ +(* libguestfs + * Copyright (C) 2014 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + *) + +(* Please read generator/README first. *) + +open Printf + +open Types +open Utils +open Pr +open Docstrings + +(* The tests in each subdirectory. *) +let tests = [ + "inspector", { + check = [ + "test-virt-inspector.sh"; + ]; + check_fast = [ + "test-xmllint.sh"; + ]; + check_slow = []; + check_local_guests = [ + "test-virt-inspector-local-guests.sh"; + ]; + check_data = [ + "example-debian-netinst-cd.xml"; + "example-debian.xml"; + "example-fedora-dvd.xml"; + "example-fedora-netinst-cd.xml"; + "example-fedora.xml"; + "example-rhel-6-dvd.xml"; + "example-rhel-6-netinst-cd.xml"; + "example-rhel-6.xml"; + "example-ubuntu-live-cd.xml"; + "example-ubuntu.xml"; + "example-windows-2003-x64-cd.xml"; + "example-windows-2003-x86-cd.xml"; + "example-windows.xml"; + "example-windows-xp-cd.xml"; + "expected-debian.img.xml"; + "expected-fedora.img.xml"; + "expected-ubuntu.img.xml"; + "expected-windows.img.xml"; + "virt-inspector.rng"; + ] + }; + +] diff --git a/generator/tests_mk.ml b/generator/tests_mk.ml new file mode 100644 index 0000000..cb7adbb --- /dev/null +++ b/generator/tests_mk.ml @@ -0,0 +1,104 @@ +(* libguestfs + * Copyright (C) 2014 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + *) + +(* Please read generator/README first. *) + +open Printf + +open Types +open Tests +open Utils +open Pr +open Docstrings + +(* Generate the tests.mk files in each subdirectory. *) +let generate_tests_mk dir tests () = + generate_header HashStyle GPLv2plus; + + let all_test_files = + tests.check @ tests.check_fast @ tests.check_slow @ tests.check_local_guests + in + + pr "localtestsdir = $(alltestsdir)/%s\n" dir; + + if tests.check_data <> [] then ( + pr "\n"; + pr "localtests_DATA ="; + List.iter (fun n -> pr " \\\n\t%s" n) tests.check_data; + pr "\n"; + ); + + pr "\n"; + pr "localtests_SCRIPTS ="; + List.iter (fun n -> pr " \\\n\t%s" n) all_test_files; + pr "\n"; + + (* Create rules so that 'make -C dir check' etc will do something. *) + if tests.check <> [] then ( + pr "\n"; + (* Note that we cannot create a simple 'check' target since + * automake will (silently) overrule it. + *) + pr "TESTS_ENVIRONMENT = $(top_builddir)/run\n"; + pr "TESTS = $(top_builddir)/test-harness\n"; + pr "\n"; + pr "check-valgrind:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --valgrind\n"; + pr "\n"; + pr "check-direct:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --direct\n"; + pr "\n"; + pr "check-valgrind-direct:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --valgrind --direct\n"; + pr "\n"; + pr "check-uml:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --uml\n"; + pr "\n"; + pr "check-valgrind-uml:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --valgrind --uml\n"; + pr "\n"; + pr "check-with-upstream-qemu:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --upstream-qemu\n"; + pr "\n"; + pr "check-with-upstream-libvirt:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --upstream-libvirt\n"; + ); + + if tests.check_fast <> [] then ( + pr "\n"; + pr "check-fast:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --fast\n" + ); + + if tests.check_slow <> [] then ( + pr "\n"; + pr "check-slow:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --slow\n" + ); + + if tests.check_local_guests <> [] then ( + pr "\n"; + pr "check-local-guests:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --local-guests\n"; + pr "\n"; + pr "check-valgrind-local-guests:\n"; + pr "\t$(top_builddir)/run $(top_builddir)/test-harness --valgrind --local-guests\n" + ); + + pr "\n"; + pr "EXTRA_DIST += tests.mk\n" diff --git a/generator/types.ml b/generator/types.ml index 63aa235..3f1f682 100644 --- a/generator/types.ml +++ b/generator/types.ml @@ -207,6 +207,15 @@ type fish_output_t = | FishOutputOctal (* for int return, print in octal *) | FishOutputHexadecimal (* for int return, print in hex *) +type test = { + check : string list; + (* "fast" here means the appliance is not needed *) + check_fast : string list; + check_slow : string list; + check_local_guests : string list; + check_data : string list; +} + (* See guestfs(3)/EXTENDING LIBGUESTFS. *) type c_api_tests = (c_api_test_init * c_api_test_prereq * c_api_test * c_api_test_cleanup) list and c_api_test = diff --git a/inspector/Makefile.am b/inspector/Makefile.am index 86e0cd7..2bf78e9 100644 --- a/inspector/Makefile.am +++ b/inspector/Makefile.am @@ -17,6 +17,8 @@ include $(top_srcdir)/subdir-rules.mk +generator_built = tests.mk + example_xml = \ example-debian.xml \ example-fedora.xml \ @@ -38,6 +40,7 @@ EXTRA_DIST = \ expected-fedora.img.xml \ expected-ubuntu.img.xml \ expected-windows.img.xml \ + test-virt-inspector-local-guests.sh \ test-virt-inspector.sh \ test-xmllint.sh.in \ virt-inspector.pod @@ -100,16 +103,4 @@ stamp-virt-inspector.pod: virt-inspector.pod $< touch $@ -TESTS_ENVIRONMENT = $(top_builddir)/run --test -TESTS = test-virt-inspector.sh -if HAVE_XMLLINT -TESTS += test-xmllint.sh -endif - -check-valgrind: - $(MAKE) TESTS="test-virt-inspector.sh" VG="$(top_builddir)/run @VG@" check - -check-valgrind-local-guests: - for g in $(GUESTS); do \ - $(top_builddir)/run --test @VG@ ./virt-inspector -c "$(libvirt_ro_uri)" -d "$$g" || exit $$?; \ - done +include $(srcdir)/tests.mk diff --git a/inspector/test-virt-inspector-local-guests.sh.in b/inspector/test-virt-inspector-local-guests.sh.in new file mode 100755 index 0000000..7eade26 --- /dev/null +++ b/inspector/test-virt-inspector-local-guests.sh.in @@ -0,0 +1,26 @@ +#!/bin/bash - +# libguestfs virt-inspector test script +# @configure_input@ +# Copyright (C) 2012-2014 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +export LANG=C +set -e +set -x + +for g in "$@"; do + virt-inspector -c "@libvirt_ro_uri@" -d "$g" || exit 1 +done diff --git a/inspector/test-virt-inspector.sh b/inspector/test-virt-inspector.sh index 6fab253..d666e7b 100755 --- a/inspector/test-virt-inspector.sh +++ b/inspector/test-virt-inspector.sh @@ -20,23 +20,16 @@ export LANG=C set -e set -x -# Allow this test to be skipped. -if [ -n "$SKIP_TEST_VIRT_INSPECTOR_SH" ]; then - echo "$0: skipping test because SKIP_TEST_VIRT_INSPECTOR_SH is set." - exit 77 -fi - # ntfs-3g can't set UUIDs right now, so ignore just that <uuid>. diff_ignore="-I <uuid>[0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F]</uuid>" -for f in ../tests/guests/{debian,fedora,ubuntu,windows}.img; do - # Ignore zero-sized windows.img if ntfs-3g is not installed. +for f in $phonydir/{debian,fedora,ubuntu,windows}.img; do if [ -s "$f" ]; then b=$(basename "$f" .xml) - $VG virt-inspector -a "$f" > "actual-$b.xml" + $VG virt-inspector -a "$f" > actual-$b.xml # This 'diff' command will fail (because of -e option) if there # are any differences. - diff -ur $diff_ignore "expected-$b.xml" "actual-$b.xml" + diff -ur $diff_ignore $srcdir/expected-$b.xml actual-$b.xml fi done diff --git a/inspector/test-xmllint.sh.in b/inspector/test-xmllint.sh.in index aef5ebc..f1195fd 100755 --- a/inspector/test-xmllint.sh.in +++ b/inspector/test-xmllint.sh.in @@ -19,6 +19,11 @@ export LANG=C set -e +if ! "@XMLLINT@" --version >/dev/null 2>&1; then + echo "$0: test skipped before xmllint is not installed" + exit 77 +fi + for f in $srcdir/example-*.xml; do @XMLLINT@ --noout --relaxng $srcdir/virt-inspector.rng $f done diff --git a/inspector/tests.mk b/inspector/tests.mk new file mode 100644 index 0000000..55004db --- /dev/null +++ b/inspector/tests.mk @@ -0,0 +1,83 @@ +# libguestfs generated file +# WARNING: THIS FILE IS GENERATED FROM: +# generator/ *.ml +# ANY CHANGES YOU MAKE TO THIS FILE WILL BE LOST. +# +# Copyright (C) 2009-2014 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +localtestsdir = $(alltestsdir)/inspector + +localtests_DATA = \ + example-debian-netinst-cd.xml \ + example-debian.xml \ + example-fedora-dvd.xml \ + example-fedora-netinst-cd.xml \ + example-fedora.xml \ + example-rhel-6-dvd.xml \ + example-rhel-6-netinst-cd.xml \ + example-rhel-6.xml \ + example-ubuntu-live-cd.xml \ + example-ubuntu.xml \ + example-windows-2003-x64-cd.xml \ + example-windows-2003-x86-cd.xml \ + example-windows.xml \ + example-windows-xp-cd.xml \ + expected-debian.img.xml \ + expected-fedora.img.xml \ + expected-ubuntu.img.xml \ + expected-windows.img.xml \ + virt-inspector.rng + +localtests_SCRIPTS = \ + test-virt-inspector.sh \ + test-xmllint.sh \ + test-virt-inspector-local-guests.sh + +TESTS_ENVIRONMENT = $(top_builddir)/run +TESTS = $(top_builddir)/test-harness + +check-valgrind: + $(top_builddir)/run $(top_builddir)/test-harness --valgrind + +check-direct: + $(top_builddir)/run $(top_builddir)/test-harness --direct + +check-valgrind-direct: + $(top_builddir)/run $(top_builddir)/test-harness --valgrind --direct + +check-uml: + $(top_builddir)/run $(top_builddir)/test-harness --uml + +check-valgrind-uml: + $(top_builddir)/run $(top_builddir)/test-harness --valgrind --uml + +check-with-upstream-qemu: + $(top_builddir)/run $(top_builddir)/test-harness --upstream-qemu + +check-with-upstream-libvirt: + $(top_builddir)/run $(top_builddir)/test-harness --upstream-libvirt + +check-fast: + $(top_builddir)/run $(top_builddir)/test-harness --fast + +check-local-guests: + $(top_builddir)/run $(top_builddir)/test-harness --local-guests + +check-valgrind-local-guests: + $(top_builddir)/run $(top_builddir)/test-harness --valgrind --local-guests + +EXTRA_DIST += tests.mk diff --git a/tests/data/Makefile.am b/tests/data/Makefile.am index d00400d..73ffccd 100644 --- a/tests/data/Makefile.am +++ b/tests/data/Makefile.am @@ -85,12 +85,14 @@ images_files_build = \ initrd-x86_64.img.gz \ test-grep.txt.gz -check_DATA = $(images_files_build) test.iso - -CLEANFILES = $(images_files_build) test.iso - images_files = $(images_files_src) $(images_files_build) +localtestsdir = $(alltestsdir)/tests/data + +localtests_DATA = $(images_files) test.iso + +CLEANFILES = $(images_files_build) test.iso + test.iso: $(images_files) rm -f $@ $@-t mkdir -p directory diff --git a/tests/guests/Makefile.am b/tests/guests/Makefile.am index 62f5d14..7cfdbd3 100644 --- a/tests/guests/Makefile.am +++ b/tests/guests/Makefile.am @@ -55,62 +55,34 @@ disk_images = \ # time and we need the tools we have built in order to make it. check_DATA = $(disk_images) guests-all-good.xml -CLEANFILES = $(check_DATA) \ - guests-all-good.xml \ - stamp-fedora-md.img \ - *.tmp.* +$(disk_images) guests-all-good.xml: stamp-guests -# Make several different blank images. These are not guests, but we -# include them in the libvirt fake XML to make sure that virt-df and -# virt-alignment-scan don't break when they encounter them. -blank-%.img: - rm -f $@ $@-t - $(top_builddir)/run \ - ../../fish/guestfish \ - -N $@-t="$$(echo $@ | sed -e 's/blank-//' -e 's/.img//')" exit - mv $@-t $@ - -# Make a (dummy) Fedora image. -fedora.img: guest-aux/make-fedora-img.pl \ - guest-aux/fedora-journal.tar.xz \ - guest-aux/fedora-name.db \ - guest-aux/fedora-packages.db - SRCDIR=$(srcdir) LAYOUT=partitions $(top_builddir)/run --test $< - -# Make a (dummy) Fedora image using md devices -fedora-md1.img fedora-md2.img: stamp-fedora-md.img - -stamp-fedora-md.img: guest-aux/make-fedora-img.pl \ - guest-aux/fedora-journal.tar.xz \ - guest-aux/fedora-name.db \ - guest-aux/fedora-packages.db - rm -f $@ - SRCDIR=$(srcdir) LAYOUT=partitions-md $(top_builddir)/run --test $< +stamp-guests: $(top_builddir)/test-harness + $(top_builddir)/run $< --make-phony-guests-only touch $@ -fedora-btrfs.img: guest-aux/make-fedora-img.pl \ - guest-aux/fedora-journal.tar.xz \ - guest-aux/fedora-name.db \ - guest-aux/fedora-packages.db - SRCDIR=$(srcdir) LAYOUT=btrfs $(top_builddir)/run --test $< +CLEANFILES = \ + $(check_DATA) \ + stamp-guests \ + *.tmp.* -# Make a (dummy) Debian image. -debian.img: guest-aux/make-debian-img.sh - SRCDIR=$(srcdir) $(top_builddir)/run --test $< - -# Make a (dummy) Ubuntu image. -ubuntu.img: guest-aux/make-ubuntu-img.sh - SRCDIR=$(srcdir) $(top_builddir)/run --test $< - -# Make a (dummy) Windows image. -windows.img: guest-aux/make-windows-img.sh \ - guest-aux/windows-software guest-aux/windows-system - SRCDIR=$(srcdir) $(top_builddir)/run --test $< - -guests-all-good.xml: guest-aux/make-guests-all-good.pl $(disk_images) - rm -f $@ $@-t - $^ > $@-t - mv $@-t $@ +# For installed tests. +localtestsdir = $(alltestsdir)/tests/guests/guest-aux +localtests_DATA = \ + guest-aux/debian-packages \ + guest-aux/debian-syslog \ + guest-aux/fedora-journal.tar.xz \ + guest-aux/fedora-name.db \ + guest-aux/fedora-packages.db \ + guest-aux/minimal-hive \ + guest-aux/windows-software \ + guest-aux/windows-system +localtests_SCRIPTS = \ + guest-aux/make-debian-img.sh \ + guest-aux/make-fedora-img.pl \ + guest-aux/make-guests-all-good.pl \ + guest-aux/make-ubuntu-img.sh \ + guest-aux/make-windows-img.sh # Since users might not have the tools needed to create this, we also # distribute these files and they are only cleaned by 'make distclean' @@ -142,11 +114,3 @@ DISTCLEANFILES = \ guest-aux/fedora-packages.db \ guest-aux/windows-software \ guest-aux/windows-system - -# Don't construct the guests in parallel. In automake 1.13, check_DATA -# was changed so it can now run in parallel, but this causes everything -# to fall over on machines with limited memory. -# -# ALSO: the guestfish rules above for making the blank-*.img files are -# NOT safe to run in parallel. -.NOTPARALLEL: diff --git a/tests/guests/guest-aux/make-debian-img.sh b/tests/guests/guest-aux/make-debian-img.sh index 95228ab..af251b4 100755 --- a/tests/guests/guest-aux/make-debian-img.sh +++ b/tests/guests/guest-aux/make-debian-img.sh @@ -82,11 +82,11 @@ upload fstab.tmp.$$ /etc/fstab write /etc/debian_version "5.0.1" write /etc/hostname "debian.invalid" -upload $SRCDIR/guest-aux/debian-packages /var/lib/dpkg/status +upload $srcdir/guest-aux/debian-packages /var/lib/dpkg/status -upload $SRCDIR/../data/bin-x86_64-dynamic /bin/ls +upload $datadir/../data/bin-x86_64-dynamic /bin/ls -upload $SRCDIR/guest-aux/debian-syslog /var/log/syslog +upload $srcdir/guest-aux/debian-syslog /var/log/syslog mkdir /boot/grub touch /boot/grub/grub.conf diff --git a/tests/guests/guest-aux/make-fedora-img.pl b/tests/guests/guest-aux/make-fedora-img.pl index 1c23b6d..95bd6e4 100755 --- a/tests/guests/guest-aux/make-fedora-img.pl +++ b/tests/guests/guest-aux/make-fedora-img.pl @@ -44,11 +44,12 @@ my $g = Sys::Guestfs->new (); my $bootdev; -foreach ('LAYOUT', 'SRCDIR') { - defined ($ENV{$_}) or die "Missing environment variable: $_"; +if (@ARGV < 1 || $ARGV[0] !~ /^--layout=(.*)/) { + die "usage: $0 --layout=(partitions|partitions-md|btrfs)\n" } +my $layout = $1; -if ($ENV{LAYOUT} eq 'partitions') { +if ($layout eq 'partitions') { push (@images, "fedora.img.tmp.$$"); open (my $fstab, '>', "fstab.tmp.$$") or die; @@ -73,7 +74,7 @@ EOF init_lvm_root ('/dev/sda2'); } -elsif ($ENV{LAYOUT} eq 'partitions-md') { +elsif ($layout eq 'partitions-md') { push (@images, "fedora-md1.img.tmp.$$", "fedora-md2.img.tmp.$$"); open (my $fstab, '>', "fstab.tmp.$$") or die; @@ -122,7 +123,7 @@ EOF init_lvm_root ('/dev/md/root'); } -elsif ($ENV{LAYOUT} eq 'btrfs') { +elsif ($layout eq 'btrfs') { push (@images, "fedora-btrfs.img.tmp.$$"); open (my $fstab, '>', "fstab.tmp.$$") or die; @@ -154,7 +155,7 @@ EOF } else { - print STDERR "$0: Unknown LAYOUT: ",$ENV{LAYOUT},"\n"; + print STDERR "$0: Unknown layout: ",$layout,"\n"; exit 1; } @@ -217,12 +218,12 @@ if (-f "mdadm.tmp.$$") { unlink ("mdadm.tmp.$$") or die; } -$g->upload ($ENV{SRCDIR}.'/guest-aux/fedora-name.db', '/var/lib/rpm/Name'); -$g->upload ($ENV{SRCDIR}.'/guest-aux/fedora-packages.db', '/var/lib/rpm/Packages'); +$g->upload ($ENV{srcdir}.'/guest-aux/fedora-name.db', '/var/lib/rpm/Name'); +$g->upload ($ENV{srcdir}.'/guest-aux/fedora-packages.db', '/var/lib/rpm/Packages'); -$g->upload ($ENV{SRCDIR}.'/../data/bin-x86_64-dynamic', '/bin/ls'); +$g->upload ($ENV{datadir}.'/../data/bin-x86_64-dynamic', '/bin/ls'); -$g->txz_in ($ENV{SRCDIR}.'/guest-aux/fedora-journal.tar.xz', '/var/log/journal'); +$g->txz_in ($ENV{srcdir}.'/guest-aux/fedora-journal.tar.xz', '/var/log/journal'); $g->mkdir ('/boot/grub'); $g->touch ('/boot/grub/grub.conf'); diff --git a/tests/guests/guest-aux/make-ubuntu-img.sh b/tests/guests/guest-aux/make-ubuntu-img.sh index 183985b..36abe55 100755 --- a/tests/guests/guest-aux/make-ubuntu-img.sh +++ b/tests/guests/guest-aux/make-ubuntu-img.sh @@ -73,9 +73,9 @@ write /etc/debian_version "5.0.1" upload release.tmp.$$ /etc/lsb-release write /etc/hostname "ubuntu.invalid" -upload $SRCDIR/guest-aux/debian-packages /var/lib/dpkg/status +upload $srcdir/guest-aux/debian-packages /var/lib/dpkg/status -upload $SRCDIR/../data/bin-x86_64-dynamic /bin/ls +upload $datadir/../data/bin-x86_64-dynamic /bin/ls mkdir /boot/grub touch /boot/grub/grub.conf diff --git a/tests/guests/guest-aux/make-windows-img.sh b/tests/guests/guest-aux/make-windows-img.sh index 575c2ee..3504658 100755 --- a/tests/guests/guest-aux/make-windows-img.sh +++ b/tests/guests/guest-aux/make-windows-img.sh @@ -59,10 +59,10 @@ mount /dev/sda2 / mkdir-p /Windows/System32/Config mkdir-p /Windows/System32/Drivers -upload $SRCDIR/guest-aux/windows-software /Windows/System32/Config/SOFTWARE -upload $SRCDIR/guest-aux/windows-system /Windows/System32/Config/SYSTEM +upload $srcdir/guest-aux/windows-software /Windows/System32/Config/SOFTWARE +upload $srcdir/guest-aux/windows-system /Windows/System32/Config/SYSTEM -upload $SRCDIR/../data/bin-win32.exe /Windows/System32/cmd.exe +upload $datadir/../data/bin-win32.exe /Windows/System32/cmd.exe mkdir "/Program Files" touch /autoexec.bat -- 2.0.4

1 0

Introducing snf-image-creator
by Nikos Skalkotos 02 Oct '14

02 Oct '14

Hello all, I'd like to introduce to you snf-image-creator, an OS image creation tool that makes heavy use of libguestfs. We have been developing it for almost 3 years and use it with the Synnefo[1] open source IaaS cloud software and also with Google's Ganeti[2] cluster management software. However it is completely independent from the above solutions and can be used as a generic image creation tool too. It comes in two variants: * snf-image-creator: A user-friendly dialog-based program * snf-mkimage: A non-interactive command-line program Unlike Oz, it takes as input, media that represents a hard disk (a block device, an image file or the host system itself) with an installed OS on it. It snapshots the hard disk, performs various cleanup tasks on the snapshot (like virt-sysprep), shrinks the snapshot and finally dumps the final prepared image on a local file or uploads and registers it with a Synnefo deployment. It can fully operate on Linux, Windows and FreeBSD input media. On Windows it can automatically install the needed VirtIO drivers and perform sysprep. It can also be used to create an image out of the Linux host system itself (host bundling process). The documentation page is here: https://www.synnefo.org/docs/snf-image-creator/latest/index.html Our code is hosted on github: https://github.com/grnet/snf-image-creator It is still under active development but it's pretty stable. We 've been using it to provide the official images for our IaaS public cloud for at least 2 years. Your feedback is more than welcome! On behalf of the Synnefo development team, Nikos [1] https://www.synnefo.org/ [2] https://code.google.com/p/ganeti/

2 1