1.39 proposal: Let's split up the libguestfs git repo and tarballs
My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test. The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace. My suggested split would be something like this: * libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website * 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby * virt-customize and related tools, we'd probably call this subproject "virt-builder". It would include virt-builder, virt-customize and virt-sysprep, since they share a lot of common code. * 1 project for each of the following items: small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg * I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them. * common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified. M4, PO, and tests would be split between the projects as appropriate. My proposal would be to do this incrementally, rather than all at once, moving the easier things out first. Thoughts? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 02/09/2018 12:01 PM, Richard W.M. Jones wrote:
My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test.
The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace.
Sounds reasonable to me as an observer. Would you also create a meta-package that has all the other projects as submodules, and which gets a new commit any time any one of the submodules does a release, to still make it easy for someone who wants to grab everything that the old monolithic repo used to provide?
* common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified.
git submodules are a pain to work with sometimes, but they do sound like the best approach for what you are documenting here. Dan Berrange's work on making keycodemapdb a submodule to multiple projects may prove to be a useful inspiration in the process. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Fri, Feb 09, 2018 at 01:07:13PM -0600, Eric Blake wrote:
On 02/09/2018 12:01 PM, Richard W.M. Jones wrote:
My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test.
The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace.
Sounds reasonable to me as an observer. Would you also create a meta-package that has all the other projects as submodules, and which gets a new commit any time any one of the submodules does a release, to still make it easy for someone who wants to grab everything that the old monolithic repo used to provide?
I guess we could although it has a danger of getting out of date if no one works on it.
* common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified.
git submodules are a pain to work with sometimes, but they do sound like the best approach for what you are documenting here. Dan Berrange's work on making keycodemapdb a submodule to multiple projects may prove to be a useful inspiration in the process.
I'm not a fan of submodules either, but in this one case I do think they would work. It's still an open question how this would translate to tarballs which realistically need to be self-contained. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Fri, Feb 09, 2018 at 06:01:53PM +0000, Richard W.M. Jones wrote:
My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test.
The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace.
My suggested split would be something like this:
* libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website
* 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby
So, the core library above would still include the API description and "make install" it into some location, such that these language bindings cna auto-generate themselves I presume. I guess that means you would rarely need to do releases of these language bindings, as one release ought to be capable of being built against multiple versions of the core library ?
* virt-customize and related tools, we'd probably call this subproject "virt-builder". It would include virt-builder, virt-customize and virt-sysprep, since they share a lot of common code.
Makes sense to have virt-builder as a thing in its own right.
* 1 project for each of the following items:
small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue)
guestfish
virt-alignment-scan and virt-df
virt-dib
virt-get-kernel
virt-resize
virt-sparsify
virt-v2v and virt-p2v
virt-win-reg
Ok
* I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them.
* common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified.
I guess sub-modules are reasonable for this, unless you actually modulized the generator itself, such that the language binding generation code could be a loadable module. That way the core generator could be in the core library (and its -devel) package, and the language binding repo could have the langauge specific plugin for the generator ? Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Feb 12, 2018 at 09:22:30AM +0000, Daniel P. Berrangé wrote:
On Fri, Feb 09, 2018 at 06:01:53PM +0000, Richard W.M. Jones wrote:
My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test.
The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace.
My suggested split would be something like this:
* libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website
* 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby
So, the core library above would still include the API description and "make install" it into some location, such that these language bindings cna auto-generate themselves I presume. I guess that means you would rarely need to do releases of these language bindings, as one release ought to be capable of being built against multiple versions of the core library ?
Certainly the language bindings are the hardest to deal with but also the most important to move out in terms of reducing dependencies. The "API description" continues to be the generator, turned into a git submodule and shared across all of them. But it's not a fully formed plan. One particular difficulty is - as you note - that some of the bindings cannot be compiled against a different version of libguestfs (we discovered this when we turned the Python bindings into a PyPi module), so likely they'd all need to be released at the same time, or else modified so at least they need a minimum version of libguestfs.
* I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them.
* common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified.
I guess sub-modules are reasonable for this, unless you actually modulized the generator itself, such that the language binding generation code could be a loadable module. That way the core generator could be in the core library (and its -devel) package, and the language binding repo could have the langauge specific plugin for the generator ?
We can keep the generator as a single program with only a small modification (it needs to check if a directory exists before putting files there). How exactly this all works when compiling from tarballs is also not clear. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Mon, Feb 12, 2018 at 12:09:23PM +0000, Richard W.M. Jones wrote:
On Mon, Feb 12, 2018 at 09:22:30AM +0000, Daniel P. Berrangé wrote:
On Fri, Feb 09, 2018 at 06:01:53PM +0000, Richard W.M. Jones wrote:
My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test.
The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace.
My suggested split would be something like this:
* libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website
* 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby
So, the core library above would still include the API description and "make install" it into some location, such that these language bindings cna auto-generate themselves I presume. I guess that means you would rarely need to do releases of these language bindings, as one release ought to be capable of being built against multiple versions of the core library ?
Certainly the language bindings are the hardest to deal with but also the most important to move out in terms of reducing dependencies. The "API description" continues to be the generator, turned into a git submodule and shared across all of them.
Is there any sense in having the generator program "make install"d by the core library, so it can be consumed by the language bindings that way ? You can use pkg-config to report custom variables to inform them where it can be found. eg libvirt uses this for its API XML description $ pkg-config --variable libvirt_api libvirt /usr/share/libvirt/api/libvirt-api.xml
But it's not a fully formed plan. One particular difficulty is - as you note - that some of the bindings cannot be compiled against a different version of libguestfs (we discovered this when we turned the Python bindings into a PyPi module), so likely they'd all need to be released at the same time, or else modified so at least they need a minimum version of libguestfs.
I don't know if you solved the problem wrt python bindings, but I think it is particularly important to be able to compile the language bindings against arbitrary versions if at all possible. Applications like OpenStack like to track the latest pypi versions, but at the same time have to work against whatever libguestfs C version the distro provides. For libvirt, we had a cut-off point at 0.9.11 as min C library version for the python to be able to build against, as that's where we started installing the API XML description in -devel packages.
* I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them.
* common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified.
I guess sub-modules are reasonable for this, unless you actually modulized the generator itself, such that the language binding generation code could be a loadable module. That way the core generator could be in the core library (and its -devel) package, and the language binding repo could have the langauge specific plugin for the generator ?
We can keep the generator as a single program with only a small modification (it needs to check if a directory exists before putting files there).
How exactly this all works when compiling from tarballs is also not clear.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Friday, 9 February 2018 19:01:53 CEST Richard W.M. Jones wrote:
My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test.
The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace.
As also other answers to this email say, splitting tools, and bindings may be very complex, and thus for now it is still a too far goal. However...
My suggested split would be something like this:
[...] virt-v2v and virt-p2v
I'd rather split virt-p2v in its own repository. There are various reasons for this: - it does not use libguestfs (the library), just the tools for testing stuff - the communication with virt-v2v is done via network, and its capabilities are dynamically probed (so theoretically virt-p2v, and virt-v2v can be used even when their versions are odd) - it is written only in C However, even if it looks simple, in reality there are number of common things used from the rest of the libguestfs tree: 1) gnulib 2) some build system bits (e.g. m4/guestfs-v2v.m4) 3) auto-cleanup bits (e.g. CLEANUP_FREE), although only few are used (CLEANUP_FREE, CLEANUP_FREE_STRING_LIST, CLEANUP_PCLOSE, CLEANUP_FCLOSE, and CLEANUP_XMLFREETEXTWRITER) 4) other internal macros, i.e. guestfs-utils.h 5) the list of credits generated by the generator (i.e. generator/authors.ml) 6) the p2v configuration generated by the generator (i.e. generator/p2v_config.ml) 7) test images/data (phony images, and virt-tools) 8) the miniexpect module, right now out of the p2v subdirectory Possible solutions may/might be: 1) add own submodule (use its own set of modules) 2) copy/implement them them locally: luckly they are not many, so inlining them in configure.ac will not be a problem; the common bits (e.g. the distro detection from os-release) can be split in its own module in libguestfs, copying it in p2v 3/4) have a local version of them; not pretty, although they are not that many 5) this list is reflected in two places: the p2v/about-authors.c file, and the AUTHORS file (theoretically mandatory for automake, unless "foreign" is used, which it is); my idea was to go back to a manually written about-authors.c file without the libguestfs credits, leaving the few p2v ones easy to manage; the same for the AUTHORS file 6) this is a bit more complex: my idea was to keep it as OCaml script to run at build time, instead of being statically shipped at dist time 7) create their own versions at test time using guestfish/virt-builder; maybe use a fedora image, instead of a phony windows one (will avoid hivex for the tests) 8) The other problem is how to split the repository, as the various bits are in different places: a) git filter-branch --subdirectory-filter p2v + very small repo with the current p2v subdirectory + preserves the history of the p2v subdirectory, with branches and tags - missing all the other bits, which will have no history - not usable to build older releases (e.g. for bisecting) b) create a work branch in libguestfs, then in that branch move/copy all the stuff making the p2v subdirectory build standalone there, and then import the content of the p2v subdirectory of that branch in a new empty repo + very small repo with the current p2v subdirectory - no history, no tags nor branches + using a graft it is possible to "stitch" the history of the new repo with the work branch in libguestfs c) git filter-branch to remove all the bits not related to p2v from all the commits + not that big repo + preserves the history of all the content, with branches and tags - will take a very long time to create (e.g. iterate over and over to find out what to remove) - not usable to build older releases (e.g. for bisecting) Thanks, -- Pino Toscano
Sorry for the late reply to this ... On Tue, Apr 30, 2019 at 06:28:01PM +0200, Pino Toscano wrote:
On Friday, 9 February 2018 19:01:53 CEST Richard W.M. Jones wrote:
My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test.
The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace.
As also other answers to this email say, splitting tools, and bindings may be very complex, and thus for now it is still a too far goal.
However...
My suggested split would be something like this:
[...] virt-v2v and virt-p2v
I'd rather split virt-p2v in its own repository. There are various reasons for this: - it does not use libguestfs (the library), just the tools for testing stuff - the communication with virt-v2v is done via network, and its capabilities are dynamically probed (so theoretically virt-p2v, and virt-v2v can be used even when their versions are odd) - it is written only in C
However, even if it looks simple, in reality there are number of common things used from the rest of the libguestfs tree: 1) gnulib
We hardly use gnulib in virt-p2v. I think it's only used for ignore-value.h, getprogname.h, and c-ctype.h, all of which are likely to be easily worked around.
2) some build system bits (e.g. m4/guestfs-v2v.m4)
Right, although this in itself should be split up, so no bad thing.
3) auto-cleanup bits (e.g. CLEANUP_FREE), although only few are used (CLEANUP_FREE, CLEANUP_FREE_STRING_LIST, CLEANUP_PCLOSE, CLEANUP_FCLOSE, and CLEANUP_XMLFREETEXTWRITER) 4) other internal macros, i.e. guestfs-utils.h
Common code is a bit tricker, as is ...
5) the list of credits generated by the generator (i.e. generator/authors.ml) 6) the p2v configuration generated by the generator (i.e. generator/p2v_config.ml)
... the generator and ...
7) test images/data (phony images, and virt-tools)
test data.
8) the miniexpect module, right now out of the p2v subdirectory
This is only used by virt-p2v I think, so it could go with virt-p2v or be made into a separate project.
Possible solutions may/might be: 1) add own submodule (use its own set of modules)
I think we should ditch gnulib as much as possible, so see above.
2) copy/implement them them locally: luckly they are not many, so inlining them in configure.ac will not be a problem; the common bits (e.g. the distro detection from os-release) can be split in its own module in libguestfs, copying it in p2v 3/4) have a local version of them; not pretty, although they are not that many 5) this list is reflected in two places: the p2v/about-authors.c file, and the AUTHORS file (theoretically mandatory for automake, unless "foreign" is used, which it is); my idea was to go back to a manually written about-authors.c file without the libguestfs credits, leaving the few p2v ones easy to manage; the same for the AUTHORS file 6) this is a bit more complex: my idea was to keep it as OCaml script to run at build time, instead of being statically shipped at dist time 7) create their own versions at test time using guestfish/virt-builder; maybe use a fedora image, instead of a phony windows one (will avoid hivex for the tests) 8)
So while I'm not a massive fan of git submodules, now that I have used them a few times with riscv stuff, they do solve a certain problem as long as they are managed carefully. I think the common code and the generator are cases where a submodule or two would work. Does this mean we need to move immediately to a submodule if just splitting virt-p2v, or copy code as you suggest? Maybe not, because you can imagine for just this project copying the code needed from the common/ directory, and creating a new "mini-generator" for the project which handles the little bits that need to be generated in virt-p2v. However in the long term if we split up everything a submodule or two does seem to make sense, so maybe we should start there?
The other problem is how to split the repository, as the various bits are in different places: a) git filter-branch --subdirectory-filter p2v + very small repo with the current p2v subdirectory + preserves the history of the p2v subdirectory, with branches and tags - missing all the other bits, which will have no history - not usable to build older releases (e.g. for bisecting)
I'm not exactly sure what this does. Is this something to do with preserving the history? TBH I don't think we need to bother with the history -- it exists still in libguestfs.git.
b) create a work branch in libguestfs, then in that branch move/copy all the stuff making the p2v subdirectory build standalone there, and then import the content of the p2v subdirectory of that branch in a new empty repo + very small repo with the current p2v subdirectory - no history, no tags nor branches + using a graft it is possible to "stitch" the history of the new repo with the work branch in libguestfs
c) git filter-branch to remove all the bits not related to p2v from all the commits + not that big repo + preserves the history of all the content, with branches and tags - will take a very long time to create (e.g. iterate over and over to find out what to remove) - not usable to build older releases (e.g. for bisecting)
Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Mon, Jun 10, 2019 at 04:35:52PM +0100, Richard W.M. Jones wrote:
Sorry for the late reply to this ...
On Tue, Apr 30, 2019 at 06:28:01PM +0200, Pino Toscano wrote:
The other problem is how to split the repository, as the various bits are in different places: a) git filter-branch --subdirectory-filter p2v + very small repo with the current p2v subdirectory + preserves the history of the p2v subdirectory, with branches and tags - missing all the other bits, which will have no history - not usable to build older releases (e.g. for bisecting)
I'm not exactly sure what this does. Is this something to do with preserving the history? TBH I don't think we need to bother with the history -- it exists still in libguestfs.git.
This is the approach we took in libvirt when splitting the python module out to its own repo. We didn't bother to make older versions build. Specifically the commands we ran were[1]: $ git clone libvirt libvirt-python $ cd libvirt-python $ git filter-branch --subdirectory-filter python --tag-name-filter cat -- --all $ git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d $ git reflog expire --expire=now --all $ git gc --prune=now I can't remember precisely what each of those steps does, but they are all about dropping as much cruft as possible from the new git repo. [1] https://www.redhat.com/archives/libvir-list/2013-September/msg00413.html Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Monday, 10 June 2019 17:35:52 CEST Richard W.M. Jones wrote:
Sorry for the late reply to this ...
On Tue, Apr 30, 2019 at 06:28:01PM +0200, Pino Toscano wrote:
On Friday, 9 February 2018 19:01:53 CEST Richard W.M. Jones wrote:
My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test.
The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace.
As also other answers to this email say, splitting tools, and bindings may be very complex, and thus for now it is still a too far goal.
However...
My suggested split would be something like this:
[...] virt-v2v and virt-p2v
I'd rather split virt-p2v in its own repository. There are various reasons for this: - it does not use libguestfs (the library), just the tools for testing stuff - the communication with virt-v2v is done via network, and its capabilities are dynamically probed (so theoretically virt-p2v, and virt-v2v can be used even when their versions are odd) - it is written only in C
However, even if it looks simple, in reality there are number of common things used from the rest of the libguestfs tree: 1) gnulib
We hardly use gnulib in virt-p2v. I think it's only used for ignore-value.h, getprogname.h, and c-ctype.h, all of which are likely to be easily worked around.
True, however for now it can stay, as it is one obstacle less for the split.
3) auto-cleanup bits (e.g. CLEANUP_FREE), although only few are used (CLEANUP_FREE, CLEANUP_FREE_STRING_LIST, CLEANUP_PCLOSE, CLEANUP_FCLOSE, and CLEANUP_XMLFREETEXTWRITER) 4) other internal macros, i.e. guestfs-utils.h
Common code is a bit tricker, as is ...
So far it is ~4K of bits of code copied, with ~9K more of straight copies of libxml2-cleanups.c + libxml2-writer-macros.h from common/utils.
5) the list of credits generated by the generator (i.e. generator/authors.ml) 6) the p2v configuration generated by the generator (i.e. generator/p2v_config.ml)
... the generator and ...
(5) is more shared with the rest, while (6) is basically p2v-only material.
7) test images/data (phony images, and virt-tools)
test data.
Luckly this is easy to recreate locally.
8) the miniexpect module, right now out of the p2v subdirectory
This is only used by virt-p2v I think, so it could go with virt-p2v or be made into a separate project.
Right, the upstream is somewhere else, so another "import from $URL" commit will not be any worse than what we have now.
Possible solutions may/might be: 1) add own submodule (use its own set of modules)
I think we should ditch gnulib as much as possible, so see above.\
Surely we can work on removing it after the split, step by step, if needed/wanted.
So while I'm not a massive fan of git submodules, now that I have used them a few times with riscv stuff, they do solve a certain problem as long as they are managed carefully. I think the common code and the generator are cases where a submodule or two would work.
TBH I've always found submodules tricky and problematic to use: - they are fixed to a certain revision (so no way to dynamically follow the branch of another repo) - the URL is the same for all the users, meaning you cannot reuse the same authenticated/secure protocols that your repo has - they create a certain burden when switching to a tag/branch/commit whose revision of a submodule is different than what is at the current branch - even more problematic when switching commit, and in the old commit a subdirectory is a real directory while in the latest HEAD is a submodule (or viceversa)
Does this mean we need to move immediately to a submodule if just splitting virt-p2v, or copy code as you suggest? Maybe not, because you can imagine for just this project copying the code needed from the common/ directory, and creating a new "mini-generator" for the project which handles the little bits that need to be generated in virt-p2v.
I'm actually solving in a different way, i.e. avoiding altogether the generator for p2v stuff.
However in the long term if we split up everything a submodule or two does seem to make sense, so maybe we should start there?
ATM I have enough work needed just to split p2v, so I'd prefer to delay this conversation to a later time...
The other problem is how to split the repository, as the various bits are in different places: a) git filter-branch --subdirectory-filter p2v + very small repo with the current p2v subdirectory + preserves the history of the p2v subdirectory, with branches and tags - missing all the other bits, which will have no history - not usable to build older releases (e.g. for bisecting)
I'm not exactly sure what this does. Is this something to do with preserving the history? TBH I don't think we need to bother with the history -- it exists still in libguestfs.git.
Yes, this is for preserving history, at least for the most important parts (the sources of p2v). -- Pino Toscano
On Mon, Jul 01, 2019 at 06:10:48PM +0200, Pino Toscano wrote:
On Monday, 10 June 2019 17:35:52 CEST Richard W.M. Jones wrote:
So while I'm not a massive fan of git submodules, now that I have used them a few times with riscv stuff, they do solve a certain problem as long as they are managed carefully. I think the common code and the generator are cases where a submodule or two would work.
TBH I've always found submodules tricky and problematic to use: - they are fixed to a certain revision (so no way to dynamically follow the branch of another repo) - the URL is the same for all the users, meaning you cannot reuse the same authenticated/secure protocols that your repo has - they create a certain burden when switching to a tag/branch/commit whose revision of a submodule is different than what is at the current branch - even more problematic when switching commit, and in the old commit a subdirectory is a real directory while in the latest HEAD is a submodule (or viceversa)
I mean, I don't disagree with any of this :-) For riscv we pinned Linux kernel and various toolchains at precise commits, and then only moved those forwards as we tested new combinations. Anyhow, whatever works.
Does this mean we need to move immediately to a submodule if just splitting virt-p2v, or copy code as you suggest? Maybe not, because you can imagine for just this project copying the code needed from the common/ directory, and creating a new "mini-generator" for the project which handles the little bits that need to be generated in virt-p2v.
I'm actually solving in a different way, i.e. avoiding altogether the generator for p2v stuff.
Hmm. There are parts of the current generator that apply to virt-p2v. Can we split those parts of the generator out to have a new generator that only applies to p2v? I find the generated config stuff useful, and in fact have a non-upstream patch to enhance it some more. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Monday, 1 July 2019 22:47:32 CEST Richard W.M. Jones wrote:
Does this mean we need to move immediately to a submodule if just splitting virt-p2v, or copy code as you suggest? Maybe not, because you can imagine for just this project copying the code needed from the common/ directory, and creating a new "mini-generator" for the project which handles the little bits that need to be generated in virt-p2v.
I'm actually solving in a different way, i.e. avoiding altogether the generator for p2v stuff.
Hmm. There are parts of the current generator that apply to virt-p2v.
Which ones? As I explained already, I found only two: - p2v_config.ml, which is entirely p2v-specific; when converted to another way, it can be dropped without affecting the rest of the generated sources of libguestfs - authors.ml, shared with libguestfs to generate the top-level AUTHORS file; I did not work on that yet -- my idea for it is to simply provide a static AUTHORS file for p2v, and generate the C source with the authors (for the GTK about dialog) using a simple Perl script
Can we split those parts of the generator out to have a new generator that only applies to p2v? I find the generated config stuff useful, and in fact have a non-upstream patch to enhance it some more.
While I find the generator great for all the repetitive stuff related to bindings and actions, I'd rather not statically generate files at "dist" time if possible. For libguestfs still makes sense because: - it allows users to build even the C library without OCaml installed - generating the doc texts of the APIs takes "a lot", even 2 minutes on fast machines, because of all the pod2text invocations - few files (AUTHORS, gobject/Makefile.inc) are needed to create a dist tarball OTOH, to me this static generation has few drawbacks: - generated files in dist tarballs, which makes it more tricky to patch them - a slightly different workflow than other generated files (say C sources from flex/bison/gperf) In the case of p2v, I simply do not see the need for OCaml, neither at development time nor at simple tarball building time, and not even for a static generator in general. The generated files (with the notable exception of the p2v-config.h file) can be quickly generated at build time, with no drawback needed. -- Pino Toscano
On Tue, Jul 02, 2019 at 09:32:26AM +0200, Pino Toscano wrote:
On Monday, 1 July 2019 22:47:32 CEST Richard W.M. Jones wrote:
Does this mean we need to move immediately to a submodule if just splitting virt-p2v, or copy code as you suggest? Maybe not, because you can imagine for just this project copying the code needed from the common/ directory, and creating a new "mini-generator" for the project which handles the little bits that need to be generated in virt-p2v.
I'm actually solving in a different way, i.e. avoiding altogether the generator for p2v stuff.
Hmm. There are parts of the current generator that apply to virt-p2v.
Which ones? As I explained already, I found only two: - p2v_config.ml, which is entirely p2v-specific; when converted to another way, it can be dropped without affecting the rest of the generated sources of libguestfs - authors.ml, shared with libguestfs to generate the top-level AUTHORS file; I did not work on that yet -- my idea for it is to simply provide a static AUTHORS file for p2v, and generate the C source with the authors (for the GTK about dialog) using a simple Perl script
Yes, those are the bits I mean.
Can we split those parts of the generator out to have a new generator that only applies to p2v? I find the generated config stuff useful, and in fact have a non-upstream patch to enhance it some more.
While I find the generator great for all the repetitive stuff related to bindings and actions, I'd rather not statically generate files at "dist" time if possible. For libguestfs still makes sense because: - it allows users to build even the C library without OCaml installed - generating the doc texts of the APIs takes "a lot", even 2 minutes on fast machines, because of all the pod2text invocations - few files (AUTHORS, gobject/Makefile.inc) are needed to create a dist tarball
OTOH, to me this static generation has few drawbacks: - generated files in dist tarballs, which makes it more tricky to patch them - a slightly different workflow than other generated files (say C sources from flex/bison/gperf)
In the case of p2v, I simply do not see the need for OCaml, neither at development time nor at simple tarball building time, and not even for a static generator in general. The generated files (with the notable exception of the p2v-config.h file) can be quickly generated at build time, with no drawback needed.
If you've come up with a way to deal with it I'm sure it'll be fine. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
So, the difficulty of git submodules aside, we have now split off virt-v2v and virt-p2v into separate projects. I also yesterday split off the boot analysis tools into a repo which is likely to be rarely used and which I'll probably not bother to package in Fedora. https://github.com/libguestfs/libguestfs-analysis-tools Where do we go from here? I think one of three ways. Either: * (1) * All libguestfs tools, whether written in C or OCaml, are sufficiently similar to each other and so are moved into a new libguestfs-tools project. libguestfs.git would end up containing only the daemon, core library and language bindings. (Splitting out the language bindings is technically difficult because of their interdependence with the generator, and the language bindings are closely related to the API, so I don't see the point of splitting them out anyway.) Or: * (2) * Libguestfs tools in OCaml are different, and heavier-weight, from the C ones so we put those in a new project (named ...?) The libguestfs tools in C are "lightweight" in some sense so we keep those in libguestfs.git. Note that the OCaml tools are quite interdependent on each other, eg. most of them use virt-customize code. So splitting them further is difficult. Or: * (3) * Libguestfs tools in C and OCaml are different from both libguestfs and each other, and so we should move them into two new projects (again, naming suggestions welcome). I'm not especially wedded to a particular approach, but I would like to get something going on this sooner rather than later. Rich. PS. If you want to follow the rest of this lengthy thread then: https://www.redhat.com/archives/libguestfs/2018-February/thread.html#00067 https://www.redhat.com/archives/libguestfs/2019-April/thread.html#00260 https://www.redhat.com/archives/libguestfs/2019-June/thread.html#00095 -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Fri, Nov 29, 2019 at 12:09:47PM +0000, Richard W.M. Jones wrote:
So, the difficulty of git submodules aside, we have now split off virt-v2v and virt-p2v into separate projects.
I also yesterday split off the boot analysis tools into a repo which is likely to be rarely used and which I'll probably not bother to package in Fedora. https://github.com/libguestfs/libguestfs-analysis-tools
Where do we go from here?
I think one of three ways. Either:
* (1) * All libguestfs tools, whether written in C or OCaml, are sufficiently similar to each other and so are moved into a new libguestfs-tools project.
From your original mail I see this
small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg Most of these are fairly low level core tools just exposing some libguestfs APIs / functionality in a slightly more convenient way than guestfish, whether in C or OCaml. v2v & p2v are full applications in their own right & have split already which makes alot of sense. The other one here that feels like an application, as opposed to a convience wrapper, is virt-dib. So I think perhaps that one could be a separate repo. Everything else feels fine to be bundled as one to me.
libguestfs.git would end up containing only the daemon, core library and language bindings. (Splitting out the language bindings is technically difficult because of their interdependence with the generator, and the language bindings are closely related to the API, so I don't see the point of splitting them out anyway.)
What about "guestfish" ? Do you consider that "core" or an add-on tool ? To me it feels like a core thing you'd expect to always get when you have libguestfs, so perhaps that should be in the main repo rather than tools repo.
Or:
* (2) * Libguestfs tools in OCaml are different, and heavier-weight, from the C ones so we put those in a new project (named ...?) The libguestfs tools in C are "lightweight" in some sense so we keep those in libguestfs.git.
Users shouldn't know or care what language the tool uses, so exposing an arbitrary split of tools into 2 groups based on language feels undesirable. Doubly undesirable if there's a chance some of those C tools will get ported to OCaml.
* (3) * Libguestfs tools in C and OCaml are different from both libguestfs and each other, and so we should move them into two new projects (again, naming suggestions welcome).
Same thoughts as option (2).
I'm not especially wedded to a particular approach, but I would like to get something going on this sooner rather than later.
To me it looks like something close to (1) is most sensible. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Nov 29, 2019 at 12:24:06PM +0000, Daniel P. Berrangé wrote:
On Fri, Nov 29, 2019 at 12:09:47PM +0000, Richard W.M. Jones wrote:
* (1) * All libguestfs tools, whether written in C or OCaml, are sufficiently similar to each other and so are moved into a new libguestfs-tools project.
From your original mail I see this
small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue)
guestfish
Some of the difficulty here is about what can be easily split from other things. The generator in particular (since it's in libguestfs.git) ties various things into the core repository: https://github.com/libguestfs/libguestfs/blob/355bed63f46a0e5c15aa9cbaa7ee35... In fact I've now noticed/remembered that guestfish contains a significant amount of generated code, which makes sense since it is essentially the language binding for shell script. This argues for guestfish staying in the libguestfs repo.
virt-alignment-scan and virt-df
virt-dib
virt-get-kernel
virt-resize
virt-sparsify
[virt-v2v and virt-p2v - already split out]
virt-win-reg
Most of these are fairly low level core tools just exposing some libguestfs APIs / functionality in a slightly more convenient way than guestfish, whether in C or OCaml.
v2v & p2v are full applications in their own right & have split already which makes alot of sense.
The other one here that feels like an application, as opposed to a convience wrapper, is virt-dib. So I think perhaps that one could be a separate repo. Everything else feels fine to be bundled as one to me.
virt-dib is almost independent from everything else, so it might be split into its own repo. OTOH splitting it first into a tools repo doesn't preclude further splitting later. This is really one for Pino to decide on. virt-builder is also definitely an application. However as it depends on virt-customize it could only really be moved together with virt-customize (and virt-sysprep for the same reason). Again, splitting these initially into a tools project doesn't prevent further splitting in future. virt-win-reg is written in Perl so it's the odd one out ... There are also some legacy tools written in Perl (virt-list-filesystems, virt-list-partitions, virt-tar). They haven't been packaged in Fedora for many years (not since April 2015 in fact) so I may drop them.
libguestfs.git would end up containing only the daemon, core library and language bindings. (Splitting out the language bindings is technically difficult because of their interdependence with the generator, and the language bindings are closely related to the API, so I don't see the point of splitting them out anyway.)
What about "guestfish" ? Do you consider that "core" or an add-on tool ? To me it feels like a core thing you'd expect to always get when you have libguestfs, so perhaps that should be in the main repo rather than tools repo.
Yes I think it's core - see above.
Or:
* (2) * Libguestfs tools in OCaml are different, and heavier-weight, from the C ones so we put those in a new project (named ...?) The libguestfs tools in C are "lightweight" in some sense so we keep those in libguestfs.git.
Users shouldn't know or care what language the tool uses, so exposing an arbitrary split of tools into 2 groups based on language feels undesirable. Doubly undesirable if there's a chance some of those C tools will get ported to OCaml.
* (3) * Libguestfs tools in C and OCaml are different from both libguestfs and each other, and so we should move them into two new projects (again, naming suggestions welcome).
Same thoughts as option (2).
Agreed.
I'm not especially wedded to a particular approach, but I would like to get something going on this sooner rather than later.
To me it looks like something close to (1) is most sensible.
Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Friday, 29 November 2019 13:09:47 CET Richard W.M. Jones wrote:
So, the difficulty of git submodules aside, we have now split off virt-v2v and virt-p2v into separate projects.
I also yesterday split off the boot analysis tools into a repo which is likely to be rarely used and which I'll probably not bother to package in Fedora. https://github.com/libguestfs/libguestfs-analysis-tools
Where do we go from here?
Before going on with splitting, I'd take a "pause" to review the status of the projects, after their split. Considering the work that was already done to split them, and various difficulties we faced, there is enough material to check what still need to be improved/fixed, as most probably the same issues will happen in case of further splits. The biggest issue I see so far is the impossibility to build virt-v2v (the only split project that requires libguestfs) with an uninstalled version of libguestfs. I see (at least) two problems. 1) building C programs. Usually this means having a .pc file for pkg-config in PKG_CONFIG_PATH, with headers and library in the location of that file. An approach I've seen done is to create a -uninstalled.pc file with its paths pointing to the source/build directories, and making sure the path of that -uninstalled.pc file is in PKG_CONFIG_PATH (which ./run can do). I never tried to do this, I can give it a try on Monday to see how it works. 2) building OCaml programs. It seems like ocamlfind.findlib has the possibility to locate modules in non-standard directories (see findlib.conf(5)), however I never tried that. Most probably a certain file structure is expected, which is definitely not what we have right now. IMHO solving (1) is definitely a must, otherwise there is no way to test new APIs outside of the repository with libguestfs (the library). Possibly also (2) is a must -- for example, earlier I added to the OCaml binding this: https://github.com/libguestfs/libguestfs/commit/4aa4edf972bc8df8869cdcaa4ab4... the idea is to use it in virt-v2v (not for an important change anyway), however right now it is not possible. Thanks, -- Pino Toscano
On Fri, Nov 29, 2019 at 05:04:14PM +0100, Pino Toscano wrote:
On Friday, 29 November 2019 13:09:47 CET Richard W.M. Jones wrote:
So, the difficulty of git submodules aside, we have now split off virt-v2v and virt-p2v into separate projects.
I also yesterday split off the boot analysis tools into a repo which is likely to be rarely used and which I'll probably not bother to package in Fedora. https://github.com/libguestfs/libguestfs-analysis-tools
Where do we go from here?
Before going on with splitting, I'd take a "pause" to review the status of the projects, after their split. Considering the work that was already done to split them, and various difficulties we faced, there is enough material to check what still need to be improved/fixed, as most probably the same issues will happen in case of further splits.
The biggest issue I see so far is the impossibility to build virt-v2v (the only split project that requires libguestfs) with an uninstalled version of libguestfs. I see (at least) two problems.
1) building C programs. Usually this means having a .pc file for pkg-config in PKG_CONFIG_PATH, with headers and library in the location of that file. An approach I've seen done is to create a -uninstalled.pc file with its paths pointing to the source/build directories, and making sure the path of that -uninstalled.pc file is in PKG_CONFIG_PATH (which ./run can do). I never tried to do this, I can give it a try on Monday to see how it works.
This is what libvirt has done - though we didn't call it "-uninstalled". We just have built two libvirt.pc files in different directories, one for use when installed, and one for use against uninstalled trees. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Nov 29, 2019 at 05:04:14PM +0100, Pino Toscano wrote:
On Friday, 29 November 2019 13:09:47 CET Richard W.M. Jones wrote:
So, the difficulty of git submodules aside, we have now split off virt-v2v and virt-p2v into separate projects.
I also yesterday split off the boot analysis tools into a repo which is likely to be rarely used and which I'll probably not bother to package in Fedora. https://github.com/libguestfs/libguestfs-analysis-tools
Where do we go from here?
Before going on with splitting, I'd take a "pause" to review the status of the projects, after their split. Considering the work that was already done to split them, and various difficulties we faced, there is enough material to check what still need to be improved/fixed, as most probably the same issues will happen in case of further splits.
The biggest issue I see so far is the impossibility to build virt-v2v (the only split project that requires libguestfs) with an uninstalled version of libguestfs. I see (at least) two problems.
1) building C programs. Usually this means having a .pc file for pkg-config in PKG_CONFIG_PATH, with headers and library in the location of that file. An approach I've seen done is to create a -uninstalled.pc file with its paths pointing to the source/build directories, and making sure the path of that -uninstalled.pc file is in PKG_CONFIG_PATH (which ./run can do). I never tried to do this, I can give it a try on Monday to see how it works.
2) building OCaml programs. It seems like ocamlfind.findlib has the possibility to locate modules in non-standard directories (see findlib.conf(5)), however I never tried that. Most probably a certain file structure is expected, which is definitely not what we have right now.
IMHO solving (1) is definitely a must, otherwise there is no way to test new APIs outside of the repository with libguestfs (the library). Possibly also (2) is a must -- for example, earlier I added to the OCaml binding this: https://github.com/libguestfs/libguestfs/commit/4aa4edf972bc8df8869cdcaa4ab4... the idea is to use it in virt-v2v (not for an important change anyway), however right now it is not possible.
All good points. For OCaml I believe we can set OCAMLPATH (see also findlib.conf(5) for various other environment variables) and that will allow virt-v2v to do: ../libguestfs/run ./configure ../libguestfs/run make Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
participants (4)
-
Daniel P. Berrangé -
Eric Blake -
Pino Toscano -
Richard W.M. Jones