I had a think about what to do about tests. I didn't come to any conclusions :-( but here are some notes anyway. The problem ----------- At the moment you can run the tests by doing 'make check'. This uses the automake test framework which sucks greatly (like the rest of automake) but is slightly better than the alternatives (like the rest of automake). What we want to do is to make the guests installable. You would install a package 'libguestfs-tests', run some script from that package, and it would run the tests on the installed copy of libguestfs. This lets you separate building and testing into two separate tasks, which is useful for Koji -- the current build + test can take hours. It would let us do testing in more places, for example, building the package in Koji and then testing it on baremetal -- currently not possible with Koji. We would also like to run the tests after other packages have changed, especially qemu & the kernel -- tieing the tests into the build makes this difficult. Other issues ------------ We still want 'make check' to work, and it should always be possible to run the tests as non-root even when installed. Tests have dependencies between them. It should be possible to run the tests in parallel, where dependencies permit. There should be a flexible way to skip tests which doesn't require every test to parse a $SKIP_* environment variable, which is what happens now. The current system keeps many tests in the same directory as the program being tested, eg. fish/ contains guestfish tests. That's a good thing, I think. At the same time we have a tests/ directory that has non-specific tests, and I think we should keep that too. Language binding tests will probably have to stay as they are now, since the tests use language-specific test harnesses (eg. rubygem-minitest). It shouldn't require installing any other software. It would be nice if tests could generate temporary files without needing to give them unique names. Background reading ------------------ I'm quite intrigued by Colin's "InstalledTests" plan for GNOME: https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests There are also some relevant links at the bottom of that page. He is certainly trying to solve some of the same problems. Automake installable tests -------------------------- I think the make the tests installable, we are surely going to need to write automake rules like: testsdir = $(libdir)/guestfs/tests/fish fish_testsdir = $(testsdir)/fish fish_tests_SCRIPTS = test-a.sh test-add-domain.sh #etc fish_tests_DATA = # any data files required for the test This creates a directory of test scripts, but it doesn't solve how to run them or how to express dependencies between them. We could drop a Makefile into the test directory. How to generate that? We could have some sort of script as a test harness. How do we ensure 'make check' still works? Obvious plan is to have a check-local rule that does: check-local: make install testsdir=$(top_builddir)/tmp/check/ cd $(top_builddir)/tmp/check && run-the-tests That gets ugly if you want to allow the user to run a single subdirectory of tests, or to run single tests (something I often want to do). It seems like we need to have some kind of program to manage tests. gnome-desktop-testing-runner is a contender here (https://git.gnome.org/browse/gnome-desktop-testing/tree/src/gnome-desktop...) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org