10:35 a.m.
Sorry for the late reply to this ...
On Tue, Apr 30, 2019 at 06:28:01PM +0200, Pino Toscano wrote:
On Friday, 9 February 2018 19:01:53 CEST Richard W.M. Jones wrote:
> My contention is that the libguestfs git repository is too large and
> unwieldy. There are too many separate, unrelated projects and as a
> result of that the source has too many dependencies and takes too long
> to build and test.
>
> The project divides (sort of) naturally into layers -- the library,
> the bindings, the various virt tools -- and could be split along those
> lines into separate projects which can then be released and evolve at
> their own pace.
As also other answers to this email say, splitting tools, and bindings
may be very complex, and thus for now it is still a too far goal.
However...
> My suggested split would be something like this:
>
> [...]
> virt-v2v and virt-p2v
I'd rather split virt-p2v in its own repository. There are various
reasons for this:
- it does not use libguestfs (the library), just the tools for testing
stuff
- the communication with virt-v2v is done via network, and its
capabilities are dynamically probed (so theoretically virt-p2v, and
virt-v2v can be used even when their versions are odd)
- it is written only in C
However, even if it looks simple, in reality there are number of common
things used from the rest of the libguestfs tree:
1) gnulib
We hardly use gnulib in virt-p2v. I think it's only used for
ignore-value.h, getprogname.h, and c-ctype.h, all of which are likely
to be easily worked around.
2) some build system bits (e.g. m4/guestfs-v2v.m4)
Right, although this in itself should be split up, so no bad thing.
3) auto-cleanup bits (e.g. CLEANUP_FREE), although only few are used
(CLEANUP_FREE, CLEANUP_FREE_STRING_LIST, CLEANUP_PCLOSE,
CLEANUP_FCLOSE, and CLEANUP_XMLFREETEXTWRITER)
4) other internal macros, i.e. guestfs-utils.h
Common code is a bit tricker, as is ...
5) the list of credits generated by the generator
(i.e. generator/authors.ml)
6) the p2v configuration generated by the generator
(i.e. generator/p2v_config.ml)
... the generator and ...
7) test images/data (phony images, and virt-tools)
test data.
8) the miniexpect module, right now out of the p2v subdirectory
This is only used by virt-p2v I think, so it could go with virt-p2v or
be made into a separate project.
Possible solutions may/might be:
1) add own submodule (use its own set of modules)
I think we should ditch gnulib as much as possible, so see above.
2) copy/implement them them locally: luckly they are not many, so
inlining them in configure.ac will not be a problem; the common
bits (e.g. the distro detection from os-release) can be split in
its own module in libguestfs, copying it in p2v
3/4) have a local version of them; not pretty, although they are not
that many
5) this list is reflected in two places: the p2v/about-authors.c file,
and the AUTHORS file (theoretically mandatory for automake, unless
"foreign" is used, which it is); my idea was to go back to a manually
written about-authors.c file without the libguestfs credits, leaving
the few p2v ones easy to manage; the same for the AUTHORS file
6) this is a bit more complex: my idea was to keep it as OCaml script
to run at build time, instead of being statically shipped at dist
time
7) create their own versions at test time using guestfish/virt-builder;
maybe use a fedora image, instead of a phony windows one (will avoid
hivex for the tests)
8)
So while I'm not a massive fan of git submodules, now that I have used
them a few times with riscv stuff, they do solve a certain problem as
long as they are managed carefully. I think the common code and the
generator are cases where a submodule or two would work.
Does this mean we need to move immediately to a submodule if just
splitting virt-p2v, or copy code as you suggest? Maybe not, because
you can imagine for just this project copying the code needed from the
common/ directory, and creating a new "mini-generator" for the project
which handles the little bits that need to be generated in virt-p2v.
However in the long term if we split up everything a submodule or two
does seem to make sense, so maybe we should start there?
The other problem is how to split the repository, as the various
bits
are in different places:
a) git filter-branch --subdirectory-filter p2v
+ very small repo with the current p2v subdirectory
+ preserves the history of the p2v subdirectory, with branches and tags
- missing all the other bits, which will have no history
- not usable to build older releases (e.g. for bisecting)
I'm not exactly sure what this does. Is this something to do with
preserving the history? TBH I don't think we need to bother with the
history -- it exists still in libguestfs.git.
b) create a work branch in libguestfs, then in that branch move/copy
all
the stuff making the p2v subdirectory build standalone there, and then
import the content of the p2v subdirectory of that branch in a new empty
repo
+ very small repo with the current p2v subdirectory
- no history, no tags nor branches
+ using a graft it is possible to "stitch" the history of the new repo
with the work branch in libguestfs
c) git filter-branch to remove all the bits not related to p2v from all
the commits
+ not that big repo
+ preserves the history of all the content, with branches and tags
- will take a very long time to create (e.g. iterate over and over to
find out what to remove)
- not usable to build older releases (e.g. for bisecting)
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/