Running lcitool in nbdkit
Am I doing this wrong? [see log below] This is using the upstream libvirt-ci checked out just now from gitlab, but it seems as if nbdkit has to be listed as a project under that repo. "./lcitool projects" shows libnbd but not nbdkit. A second question, since the libvirt-ci project does a bunch of stuff with ansible & containers, regenerating the files like this doesn't require ansible & containers? (I don't want to run those on my development machine.) $ ../libvirt-ci/lcitool manifest ci/manifest.yml Generating ci/containers/almalinux-8.Dockerfile [ERROR]: An unexpected error occurred Traceback (most recent call last): File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/projects.py", line 121, in get_packages obj = self.projects[proj] KeyError: 'nbdkit' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/rjones/d/libvirt-ci/guests/lcitool/bin/lcitool", line 36, in <module> Application().run(args) File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/application.py", line 392, in run args.func(self, args) File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/application.py", line 387, in _action_manifest manifest.generate(args.dry_run) File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/manifest.py", line 149, in generate generated = self._generate_containers(dryrun) File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/manifest.py", line 200, in _generate_containers return self._generate_formatter(dryrun, File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/manifest.py", line 191, in _generate_formatter payload = formatter.format(target, File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/formatters.py", line 482, in format facts, cross_arch, varmap = self._generator_prepare(target, File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/formatters.py", line 138, in _generator_prepare varmap = self._generator_build_varmap(facts, File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/formatters.py", line 163, in _generator_build_varmap self)._generator_build_varmap(facts, File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/formatters.py", line 87, in _generator_build_varmap pkgs.update(projects.get_packages(selected_projects, facts, cross_arch)) File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/projects.py", line 123, in get_packages obj = self.internal_projects[proj] KeyError: 'nbdkit' Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Sat, Mar 12, 2022 at 10:50:21AM +0000, Richard W.M. Jones wrote:
Am I doing this wrong? [see log below] This is using the upstream libvirt-ci checked out just now from gitlab, but it seems as if nbdkit has to be listed as a project under that repo.
"./lcitool projects" shows libnbd but not nbdkit.
Indeed, there's only libnbd there. Presumably nbdkit is something Martin hasn't got around to adding ? With recentish changes though there's no longer any need to add projects directly to libvirt-ci.git. They can be kept locally in the project eg nbdkit.git/ci/projects/nbdkit.yml then use 'lcitool --data-dir ci' and it'll find the nbdkit.yml when searching for projects. libvirt-ci.git will only need modifying it distro package mappings need additions/changes.
A second question, since the libvirt-ci project does a bunch of stuff with ansible & containers, regenerating the files like this doesn't require ansible & containers? (I don't want to run those on my development machine.)
ansible is only used when managing VMs, no need for it with containers. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Mar 14, 2022 at 09:18:04AM +0000, Daniel P. Berrangé wrote:
On Sat, Mar 12, 2022 at 10:50:21AM +0000, Richard W.M. Jones wrote:
Am I doing this wrong? [see log below] This is using the upstream libvirt-ci checked out just now from gitlab, but it seems as if nbdkit has to be listed as a project under that repo.
"./lcitool projects" shows libnbd but not nbdkit.
Indeed, there's only libnbd there. Presumably nbdkit is something Martin hasn't got around to adding ?
With recentish changes though there's no longer any need to add projects directly to libvirt-ci.git. They can be kept locally in the project eg
nbdkit.git/ci/projects/nbdkit.yml
then use 'lcitool --data-dir ci' and it'll find the nbdkit.yml when searching for projects.
libvirt-ci.git will only need modifying it distro package mappings need additions/changes.
I copied libvirt-ci.git/guests/lcitool/lcitool/ansible/vars/projects/libnbd.yml to ci/projects/nbdkit.yml. I'll probably have to adjust the packages in that file, but it did allow me to re-run the command above and regenerate the templates. Martin - I guess you must have created that nbdkit.yml file at some point, but I can't find it in your fork.
A second question, since the libvirt-ci project does a bunch of stuff with ansible & containers, regenerating the files like this doesn't require ansible & containers? (I don't want to run those on my development machine.)
ansible is only used when managing VMs, no need for it with containers.
I meant would the "manifest" subcommand do anything weird (ie. not just updating the templates), but it seems fine. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Mon, Mar 14, 2022 at 10:01:46AM +0000, Richard W.M. Jones wrote:
On Mon, Mar 14, 2022 at 09:18:04AM +0000, Daniel P. Berrangé wrote:
On Sat, Mar 12, 2022 at 10:50:21AM +0000, Richard W.M. Jones wrote:
Am I doing this wrong? [see log below] This is using the upstream libvirt-ci checked out just now from gitlab, but it seems as if nbdkit has to be listed as a project under that repo.
"./lcitool projects" shows libnbd but not nbdkit.
Indeed, there's only libnbd there. Presumably nbdkit is something Martin hasn't got around to adding ?
With recentish changes though there's no longer any need to add projects directly to libvirt-ci.git. They can be kept locally in the project eg
nbdkit.git/ci/projects/nbdkit.yml
then use 'lcitool --data-dir ci' and it'll find the nbdkit.yml when searching for projects.
libvirt-ci.git will only need modifying it distro package mappings need additions/changes.
I copied libvirt-ci.git/guests/lcitool/lcitool/ansible/vars/projects/libnbd.yml to ci/projects/nbdkit.yml. I'll probably have to adjust the packages in that file, but it did allow me to re-run the command above and regenerate the templates.
Martin - I guess you must have created that nbdkit.yml file at some point, but I can't find it in your fork.
A second question, since the libvirt-ci project does a bunch of stuff with ansible & containers, regenerating the files like this doesn't require ansible & containers? (I don't want to run those on my development machine.)
ansible is only used when managing VMs, no need for it with containers.
I meant would the "manifest" subcommand do anything weird (ie. not just updating the templates), but it seems fine.
Yep, the manifest command doesn't do anything with VMs or ansible. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Mar 14, 2022 at 10:01:46AM +0000, Richard W.M. Jones wrote:
Martin - I guess you must have created that nbdkit.yml file at some point, but I can't find it in your fork.
Found it: https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2 I guess we're still missing almalinux support in libvirt-ci though. $ ../libvirt-ci/lcitool manifest ci/manifest.yml [ERROR]: An unexpected error occurred Traceback (most recent call last): File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/manifest.py", line 87, in _normalize facts = inventory.target_facts[target] KeyError: 'almalinux-8' Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Mon, Mar 14, 2022 at 10:14:37AM +0000, Richard W.M. Jones wrote:
On Mon, Mar 14, 2022 at 10:01:46AM +0000, Richard W.M. Jones wrote:
Martin - I guess you must have created that nbdkit.yml file at some point, but I can't find it in your fork.
Found it:
https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2
I guess we're still missing almalinux support in libvirt-ci though.
$ ../libvirt-ci/lcitool manifest ci/manifest.yml [ERROR]: An unexpected error occurred Traceback (most recent call last): File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/manifest.py", line 87, in _normalize facts = inventory.target_facts[target] KeyError: 'almalinux-8'
Current upstream has almalinux-8, so I think this nbdkit2 branch is just a little outdated. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
I rebased Martin's tree[1] on top of libvirt-ci head, and reran lcitool. There are several places where I changed the generated files in nbdkit.git and where those changes are overwritten (I didn't realise those files were generated). I think I can fix most of them. But a couple of larger problems I can see ... (a) lcitool wants to make this change: diff --git a/ci/containers/fedora-rawhide.Dockerfile b/ci/containers/fedora-rawhide.Dockerfile index 7e0a8d25..d4409f87 100644 --- a/ci/containers/fedora-rawhide.Dockerfile +++ b/ci/containers/fedora-rawhide.Dockerfile @@ -6,10 +6,7 @@ FROM registry.fedoraproject.org/fedora:rawhide -RUN dnf update -y && \ - dnf install 'dnf-command(config-manager)' -y && \ - dnf config-manager -y --add-repo=http://dl.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/fedora-rawhide-ke... && \ - dnf update -y --nogpgcheck fedora-gpg-keys && \ +RUN dnf update -y --nogpgcheck fedora-gpg-keys && \ I added "dnf install 'dnf-command(config-manager)'" in: https://gitlab.com/nbdkit/nbdkit/-/commit/e359fab8c682436faff914591b5ff42e29... because the dnf config-manager command doesn't work. The dnf config-manager command in turn was added earlier in Martin's original commit. I don't know if Martin added that manually or if an earlier version of lcitool generated it, but it is necessary in order to use the nodebug kernel, and using the nodebug kernel is necessary to avoid performance problems, timeouts and out of memory problems when using libguestfs on Rawhide. (b) lcitool wants to remove my perl conditionalization changes: @@ -312,11 +309,10 @@ mingw32-fedora-35: - mingw32-fedora-35-container allow_failure: false variables: - NAME: fedora-35 CROSS: mingw32 - PERL: skip - RUST: skip GOLANG: skip + NAME: fedora-35 + RUST: skip which I added here: https://gitlab.com/nbdkit/nbdkit/-/commit/414ee371ee2b1a12c6d45466d041322b24... I'm not clear where I should have added those. The rust conditionalization stuff doesn't seem to be part of lcitool so I guess it's generated from another file somewhere. Be nice if original files were called "file.in" or something like that so we can tell what files are generated and what are originals. Rich. [1] https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2 -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Mon, Mar 14, 2022 at 10:41:05AM +0000, Richard W.M. Jones wrote:
I rebased Martin's tree[1] on top of libvirt-ci head, and reran lcitool. There are several places where I changed the generated files in nbdkit.git and where those changes are overwritten (I didn't realise those files were generated). I think I can fix most of them.
But a couple of larger problems I can see ...
(a) lcitool wants to make this change:
diff --git a/ci/containers/fedora-rawhide.Dockerfile b/ci/containers/fedora-rawhide.Dockerfile index 7e0a8d25..d4409f87 100644 --- a/ci/containers/fedora-rawhide.Dockerfile +++ b/ci/containers/fedora-rawhide.Dockerfile @@ -6,10 +6,7 @@
FROM registry.fedoraproject.org/fedora:rawhide
-RUN dnf update -y && \ - dnf install 'dnf-command(config-manager)' -y && \ - dnf config-manager -y --add-repo=http://dl.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/fedora-rawhide-ke... && \ - dnf update -y --nogpgcheck fedora-gpg-keys && \ +RUN dnf update -y --nogpgcheck fedora-gpg-keys && \
I added "dnf install 'dnf-command(config-manager)'" in: https://gitlab.com/nbdkit/nbdkit/-/commit/e359fab8c682436faff914591b5ff42e29... because the dnf config-manager command doesn't work.
The dnf config-manager command in turn was added earlier in Martin's original commit. I don't know if Martin added that manually or if an earlier version of lcitool generated it, but it is necessary in order to use the nodebug kernel, and using the nodebug kernel is necessary to avoid performance problems, timeouts and out of memory problems when using libguestfs on Rawhide.
Ok, that need for an extra special repo for rawhide no-debug kerenl isn't something lcitool knows how to cope with right now. So must be been manually editted.
(b) lcitool wants to remove my perl conditionalization changes:
@@ -312,11 +309,10 @@ mingw32-fedora-35: - mingw32-fedora-35-container allow_failure: false variables: - NAME: fedora-35 CROSS: mingw32 - PERL: skip - RUST: skip GOLANG: skip + NAME: fedora-35 + RUST: skip
which I added here:
https://gitlab.com/nbdkit/nbdkit/-/commit/414ee371ee2b1a12c6d45466d041322b24...
I'm not clear where I should have added those. The rust conditionalization stuff doesn't seem to be part of lcitool so I guess it's generated from another file somewhere.
Be nice if original files were called "file.in" or something like that so we can tell what files are generated and what are originals.
The only manually editted file is ci/manifest.yml. Everything else in the ci/ directory (the ci/gitlab.yml, ci/containers/* and ci/cirrus/*) is auto-generated from what's in the manifest.yml and shouldn't be touched. So you'll need to add the PERL=skip variable to manifest.yml Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
To sum it up, I added these commits to nbdkit which converge the generated files with what lcitool now generates: https://gitlab.com/nbdkit/nbdkit/-/commit/8e99576b89b58ad09c634b96cff579dd9a... https://gitlab.com/nbdkit/nbdkit/-/commit/200a147d6a99c593204b476d3cbb08cc79... Pipeline is currently running: https://gitlab.com/nbdkit/nbdkit/-/pipelines/491516164 To do: - Upstream Martin's patches to libvirt-ci https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2 - Add support for nodebug kernels (to libvirt-ci, I guess?) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Mon, Mar 14, 2022 at 11:33:16AM +0000, Richard W.M. Jones wrote:
To sum it up, I added these commits to nbdkit which converge the generated files with what lcitool now generates:
https://gitlab.com/nbdkit/nbdkit/-/commit/8e99576b89b58ad09c634b96cff579dd9a... https://gitlab.com/nbdkit/nbdkit/-/commit/200a147d6a99c593204b476d3cbb08cc79...
Pipeline is currently running:
https://gitlab.com/nbdkit/nbdkit/-/pipelines/491516164
To do:
- Upstream Martin's patches to libvirt-ci https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2
There were two branches, both of them are very, VERY messy, hence what I meant with stuff not being ready IMO.
- Add support for nodebug kernels (to libvirt-ci, I guess?)
I wanted to add that, but after a bit of testing I found that it did not speed up the testing as much as I needed to.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
On Mon, Mar 14, 2022 at 11:33:16AM +0000, Richard W.M. Jones wrote:
Pipeline is currently running:
I had previously fixed this failure in Alpine: perl.c:45:10: fatal error: XSUB.h: No such file or directory 45 | #include <XSUB.h> | ^~~~~~~~ This file comes from the perl-dev package which is missing on Alpine. I had that fixed before (but only in the generated files) so I backed that change out accidentally in the commit I just added to nbdkit. While I remember, Martin had an alternate fix for this: https://gitlab.com/nertpinx/nbdkit/-/commit/d93302150766415aaa7381c56120ad0f... but this fix is wrong - it completely disables Perl on every platform (which is why it appears to work). The true fix is this one (to libvirt-ci). I will send it upstream separately. diff --git a/guests/lcitool/lcitool/ansible/vars/mappings.yml b/guests/lcitool/lcitool/ansible/vars/mappings.yml index c05e59c..8b0825a 100644 --- a/guests/lcitool/lcitool/ansible/vars/mappings.yml +++ b/guests/lcitool/lcitool/ansible/vars/mappings.yml @@ -1617,6 +1617,8 @@ mappings: deb: libperl-dev rpm: perl-devel pkg: perl5-devel + Alpine314: perl-dev + AlpineEdge: perl-dev MacOS: perl5-devel OpenSUSE: Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Mon, Mar 14, 2022 at 11:54:57AM +0000, Richard W.M. Jones wrote:
On Mon, Mar 14, 2022 at 11:33:16AM +0000, Richard W.M. Jones wrote:
Pipeline is currently running:
I had previously fixed this failure in Alpine:
perl.c:45:10: fatal error: XSUB.h: No such file or directory 45 | #include <XSUB.h> | ^~~~~~~~
This file comes from the perl-dev package which is missing on Alpine. I had that fixed before (but only in the generated files) so I backed that change out accidentally in the commit I just added to nbdkit.
While I remember, Martin had an alternate fix for this:
https://gitlab.com/nertpinx/nbdkit/-/commit/d93302150766415aaa7381c56120ad0f...
but this fix is wrong - it completely disables Perl on every platform (which is why it appears to work).
The true fix is this one (to libvirt-ci). I will send it upstream separately.
diff --git a/guests/lcitool/lcitool/ansible/vars/mappings.yml b/guests/lcitool/lcitool/ansible/vars/mappings.yml index c05e59c..8b0825a 100644 --- a/guests/lcitool/lcitool/ansible/vars/mappings.yml +++ b/guests/lcitool/lcitool/ansible/vars/mappings.yml @@ -1617,6 +1617,8 @@ mappings: deb: libperl-dev rpm: perl-devel pkg: perl5-devel + Alpine314: perl-dev + AlpineEdge: perl-dev
You can also set if for all Alpine versions with Alpine: perl-dev or all distros using apk package manager with apk: perl-dev.
MacOS: perl5-devel OpenSUSE:
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Mon, Mar 14, 2022 at 12:57:13PM +0100, Martin Kletzander wrote:
You can also set if for all Alpine versions with Alpine: perl-dev or all distros using apk package manager with apk: perl-dev.
I was wondering about that :-) How about this change? https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/238 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Mon, Mar 14, 2022 at 10:41:05AM +0000, Richard W.M. Jones wrote:
(b) lcitool wants to remove my perl conditionalization changes:
@@ -312,11 +309,10 @@ mingw32-fedora-35: - mingw32-fedora-35-container allow_failure: false variables: - NAME: fedora-35 CROSS: mingw32 - PERL: skip - RUST: skip GOLANG: skip + NAME: fedora-35 + RUST: skip
which I added here:
https://gitlab.com/nbdkit/nbdkit/-/commit/414ee371ee2b1a12c6d45466d041322b24...
I'm not clear where I should have added those. The rust conditionalization stuff doesn't seem to be part of lcitool so I guess it's generated from another file somewhere.
Fixed this by modifying ci/manifest.yml. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
participants (3)
-
Daniel P. Berrangé -
Martin Kletzander -
Richard W.M. Jones