4:50 a.m.
Am I doing this wrong? [see log below] This is using the upstream
libvirt-ci checked out just now from gitlab, but it seems as if nbdkit
has to be listed as a project under that repo.
"./lcitool projects" shows libnbd but not nbdkit.
A second question, since the libvirt-ci project does a bunch of stuff
with ansible & containers, regenerating the files like this doesn't
require ansible & containers? (I don't want to run those on my
development machine.)
$ ../libvirt-ci/lcitool manifest ci/manifest.yml
Generating ci/containers/almalinux-8.Dockerfile
[ERROR]: An unexpected error occurred
Traceback (most recent call last):
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/projects.py", line 121,
in get_packages
obj = self.projects[proj]
KeyError: 'nbdkit'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rjones/d/libvirt-ci/guests/lcitool/bin/lcitool", line 36, in
<module>
Application().run(args)
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/application.py", line
392, in run
args.func(self, args)
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/application.py", line
387, in _action_manifest
manifest.generate(args.dry_run)
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/manifest.py", line 149,
in generate
generated = self._generate_containers(dryrun)
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/manifest.py", line 200,
in _generate_containers
return self._generate_formatter(dryrun,
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/manifest.py", line 191,
in _generate_formatter
payload = formatter.format(target,
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/formatters.py", line
482, in format
facts, cross_arch, varmap = self._generator_prepare(target,
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/formatters.py", line
138, in _generator_prepare
varmap = self._generator_build_varmap(facts,
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/formatters.py", line
163, in _generator_build_varmap
self)._generator_build_varmap(facts,
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/formatters.py", line
87, in _generator_build_varmap
pkgs.update(projects.get_packages(selected_projects, facts, cross_arch))
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/projects.py", line 123,
in get_packages
obj = self.internal_projects[proj]
KeyError: 'nbdkit'
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
4:18 a.m.
On Sat, Mar 12, 2022 at 10:50:21AM +0000, Richard W.M. Jones wrote:
Am I doing this wrong? [see log below] This is using the upstream
libvirt-ci checked out just now from gitlab, but it seems as if nbdkit
has to be listed as a project under that repo.
"./lcitool projects" shows libnbd but not nbdkit.
Indeed, there's only libnbd there. Presumably nbdkit is something
Martin hasn't got around to adding ?
With recentish changes though there's no longer any need to add
projects directly to libvirt-ci.git. They can be kept locally
in the project eg
nbdkit.git/ci/projects/nbdkit.yml
then use 'lcitool --data-dir ci' and it'll find the nbdkit.yml
when searching for projects.
libvirt-ci.git will only need modifying it distro package
mappings need additions/changes.
A second question, since the libvirt-ci project does a bunch of
stuff
with ansible & containers, regenerating the files like this doesn't
require ansible & containers? (I don't want to run those on my
development machine.)
ansible is only used when managing VMs, no need for it with containers.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
5:01 a.m.
On Mon, Mar 14, 2022 at 09:18:04AM +0000, Daniel P. Berrangé wrote:
On Sat, Mar 12, 2022 at 10:50:21AM +0000, Richard W.M. Jones wrote:
> Am I doing this wrong? [see log below] This is using the upstream
> libvirt-ci checked out just now from gitlab, but it seems as if nbdkit
> has to be listed as a project under that repo.
>
> "./lcitool projects" shows libnbd but not nbdkit.
Indeed, there's only libnbd there. Presumably nbdkit is something
Martin hasn't got around to adding ?
With recentish changes though there's no longer any need to add
projects directly to libvirt-ci.git. They can be kept locally
in the project eg
nbdkit.git/ci/projects/nbdkit.yml
then use 'lcitool --data-dir ci' and it'll find the nbdkit.yml
when searching for projects.
libvirt-ci.git will only need modifying it distro package
mappings need additions/changes.
I copied libvirt-ci.git/guests/lcitool/lcitool/ansible/vars/projects/libnbd.yml
to ci/projects/nbdkit.yml. I'll probably have to adjust the packages
in that file, but it did allow me to re-run the command above and
regenerate the templates.
Martin - I guess you must have created that nbdkit.yml file at some
point, but I can't find it in your fork.
> A second question, since the libvirt-ci project does a bunch of
stuff
> with ansible & containers, regenerating the files like this doesn't
> require ansible & containers? (I don't want to run those on my
> development machine.)
ansible is only used when managing VMs, no need for it with containers.
I meant would the "manifest" subcommand do anything weird (ie. not
just updating the templates), but it seems fine.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
5:02 a.m.
On Mon, Mar 14, 2022 at 10:01:46AM +0000, Richard W.M. Jones wrote:
On Mon, Mar 14, 2022 at 09:18:04AM +0000, Daniel P. Berrangé wrote:
> On Sat, Mar 12, 2022 at 10:50:21AM +0000, Richard W.M. Jones wrote:
> > Am I doing this wrong? [see log below] This is using the upstream
> > libvirt-ci checked out just now from gitlab, but it seems as if nbdkit
> > has to be listed as a project under that repo.
> >
> > "./lcitool projects" shows libnbd but not nbdkit.
>
> Indeed, there's only libnbd there. Presumably nbdkit is something
> Martin hasn't got around to adding ?
>
> With recentish changes though there's no longer any need to add
> projects directly to libvirt-ci.git. They can be kept locally
> in the project eg
>
> nbdkit.git/ci/projects/nbdkit.yml
>
> then use 'lcitool --data-dir ci' and it'll find the nbdkit.yml
> when searching for projects.
>
> libvirt-ci.git will only need modifying it distro package
> mappings need additions/changes.
I copied libvirt-ci.git/guests/lcitool/lcitool/ansible/vars/projects/libnbd.yml
to ci/projects/nbdkit.yml. I'll probably have to adjust the packages
in that file, but it did allow me to re-run the command above and
regenerate the templates.
Martin - I guess you must have created that nbdkit.yml file at some
point, but I can't find it in your fork.
> > A second question, since the libvirt-ci project does a bunch of stuff
> > with ansible & containers, regenerating the files like this doesn't
> > require ansible & containers? (I don't want to run those on my
> > development machine.)
>
> ansible is only used when managing VMs, no need for it with containers.
I meant would the "manifest" subcommand do anything weird (ie. not
just updating the templates), but it seems fine.
Yep, the manifest command doesn't do anything with VMs or ansible.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
5:14 a.m.
On Mon, Mar 14, 2022 at 10:01:46AM +0000, Richard W.M. Jones wrote:
Martin - I guess you must have created that nbdkit.yml file at some
point, but I can't find it in your fork.
Found it:
https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2
I guess we're still missing almalinux support in libvirt-ci though.
$ ../libvirt-ci/lcitool manifest ci/manifest.yml
[ERROR]: An unexpected error occurred
Traceback (most recent call last):
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/manifest.py", line 87,
in _normalize
facts = inventory.target_facts[target]
KeyError: 'almalinux-8'
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
5:21 a.m.
On Mon, Mar 14, 2022 at 10:14:37AM +0000, Richard W.M. Jones wrote:
On Mon, Mar 14, 2022 at 10:01:46AM +0000, Richard W.M. Jones wrote:
> Martin - I guess you must have created that nbdkit.yml file at some
> point, but I can't find it in your fork.
Found it:
https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2
I guess we're still missing almalinux support in libvirt-ci though.
$ ../libvirt-ci/lcitool manifest ci/manifest.yml
[ERROR]: An unexpected error occurred
Traceback (most recent call last):
File "/home/rjones/d/libvirt-ci/guests/lcitool/lcitool/manifest.py", line 87,
in _normalize
facts = inventory.target_facts[target]
KeyError: 'almalinux-8'
Current upstream has almalinux-8, so I think this nbdkit2 branch is just
a little outdated.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
5:41 a.m.
I rebased Martin's tree[1] on top of libvirt-ci head, and reran
lcitool. There are several places where I changed the generated files
in nbdkit.git and where those changes are overwritten (I didn't
realise those files were generated). I think I can fix most of them.
But a couple of larger problems I can see ...
(a) lcitool wants to make this change:
diff --git a/ci/containers/fedora-rawhide.Dockerfile
b/ci/containers/fedora-rawhide.Dockerfile
index 7e0a8d25..d4409f87 100644
--- a/ci/containers/fedora-rawhide.Dockerfile
+++ b/ci/containers/fedora-rawhide.Dockerfile
@@ -6,10 +6,7 @@
FROM registry.fedoraproject.org/fedora:rawhide
-RUN dnf update -y && \
- dnf install 'dnf-command(config-manager)' -y && \
- dnf config-manager -y
--add-repo=http://dl.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/fedora-rawhide-kernel-nodebug.repo
&& \
- dnf update -y --nogpgcheck fedora-gpg-keys && \
+RUN dnf update -y --nogpgcheck fedora-gpg-keys && \
I added "dnf install 'dnf-command(config-manager)'" in:
https://gitlab.com/nbdkit/nbdkit/-/commit/e359fab8c682436faff914591b5ff42...
because the dnf config-manager command doesn't work.
The dnf config-manager command in turn was added earlier in Martin's
original commit. I don't know if Martin added that manually or if an
earlier version of lcitool generated it, but it is necessary in order
to use the nodebug kernel, and using the nodebug kernel is necessary
to avoid performance problems, timeouts and out of memory problems
when using libguestfs on Rawhide.
(b) lcitool wants to remove my perl conditionalization changes:
@@ -312,11 +309,10 @@ mingw32-fedora-35:
- mingw32-fedora-35-container
allow_failure: false
variables:
- NAME: fedora-35
CROSS: mingw32
- PERL: skip
- RUST: skip
GOLANG: skip
+ NAME: fedora-35
+ RUST: skip
which I added here:
https://gitlab.com/nbdkit/nbdkit/-/commit/414ee371ee2b1a12c6d45466d041322...
I'm not clear where I should have added those. The rust
conditionalization stuff doesn't seem to be part of lcitool so I guess
it's generated from another file somewhere.
Be nice if original files were called "file.in" or something like that
so we can tell what files are generated and what are originals.
Rich.
[1] https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
6:22 a.m.
On Mon, Mar 14, 2022 at 10:41:05AM +0000, Richard W.M. Jones wrote:
I rebased Martin's tree[1] on top of libvirt-ci head, and reran
lcitool. There are several places where I changed the generated files
in nbdkit.git and where those changes are overwritten (I didn't
realise those files were generated). I think I can fix most of them.
But a couple of larger problems I can see ...
(a) lcitool wants to make this change:
diff --git a/ci/containers/fedora-rawhide.Dockerfile
b/ci/containers/fedora-rawhide.Dockerfile
index 7e0a8d25..d4409f87 100644
--- a/ci/containers/fedora-rawhide.Dockerfile
+++ b/ci/containers/fedora-rawhide.Dockerfile
@@ -6,10 +6,7 @@
FROM registry.fedoraproject.org/fedora:rawhide
-RUN dnf update -y && \
- dnf install 'dnf-command(config-manager)' -y && \
- dnf config-manager -y
--add-repo=http://dl.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/fedora-rawhide-kernel-nodebug.repo
&& \
- dnf update -y --nogpgcheck fedora-gpg-keys && \
+RUN dnf update -y --nogpgcheck fedora-gpg-keys && \
I added "dnf install 'dnf-command(config-manager)'" in:
https://gitlab.com/nbdkit/nbdkit/-/commit/e359fab8c682436faff914591b5ff42...
because the dnf config-manager command doesn't work.
The dnf config-manager command in turn was added earlier in Martin's
original commit. I don't know if Martin added that manually or if an
earlier version of lcitool generated it, but it is necessary in order
to use the nodebug kernel, and using the nodebug kernel is necessary
to avoid performance problems, timeouts and out of memory problems
when using libguestfs on Rawhide.
Ok, that need for an extra special repo for rawhide no-debug kerenl
isn't something lcitool knows how to cope with right now. So must
be been manually editted.
(b) lcitool wants to remove my perl conditionalization changes:
@@ -312,11 +309,10 @@ mingw32-fedora-35:
- mingw32-fedora-35-container
allow_failure: false
variables:
- NAME: fedora-35
CROSS: mingw32
- PERL: skip
- RUST: skip
GOLANG: skip
+ NAME: fedora-35
+ RUST: skip
which I added here:
https://gitlab.com/nbdkit/nbdkit/-/commit/414ee371ee2b1a12c6d45466d041322...
I'm not clear where I should have added those. The rust
conditionalization stuff doesn't seem to be part of lcitool so I guess
it's generated from another file somewhere.
Be nice if original files were called "file.in" or something like that
so we can tell what files are generated and what are originals.
The only manually editted file is ci/manifest.yml. Everything else in
the ci/ directory (the ci/gitlab.yml, ci/containers/* and ci/cirrus/*)
is auto-generated from what's in the manifest.yml and shouldn't be
touched.
So you'll need to add the PERL=skip variable to manifest.yml
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
6:33 a.m.
To sum it up, I added these commits to nbdkit which converge the
generated files with what lcitool now generates:
https://gitlab.com/nbdkit/nbdkit/-/commit/8e99576b89b58ad09c634b96cff579d...
https://gitlab.com/nbdkit/nbdkit/-/commit/200a147d6a99c593204b476d3cbb08c...
Pipeline is currently running:
https://gitlab.com/nbdkit/nbdkit/-/pipelines/491516164
To do:
- Upstream Martin's patches to libvirt-ci
https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2
- Add support for nodebug kernels (to libvirt-ci, I guess?)
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
6:53 a.m.
On Mon, Mar 14, 2022 at 11:33:16AM +0000, Richard W.M. Jones wrote:
To sum it up, I added these commits to nbdkit which converge the
generated files with what lcitool now generates:
https://gitlab.com/nbdkit/nbdkit/-/commit/8e99576b89b58ad09c634b96cff579d...
https://gitlab.com/nbdkit/nbdkit/-/commit/200a147d6a99c593204b476d3cbb08c...
Pipeline is currently running:
https://gitlab.com/nbdkit/nbdkit/-/pipelines/491516164
To do:
- Upstream Martin's patches to libvirt-ci
https://gitlab.com/nertpinx/libvirt-ci/-/commits/_nbdkit2
There were two branches, both of them are very, VERY messy, hence what
I meant with stuff not being ready IMO.
- Add support for nodebug kernels (to libvirt-ci, I guess?)
I wanted to add that, but after a bit of testing I found that it did not
speed up the testing as much as I needed to.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
_______________________________________________
Libguestfs mailing list
Libguestfs(a)redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
6:54 a.m.
On Mon, Mar 14, 2022 at 11:33:16AM +0000, Richard W.M. Jones wrote:
Pipeline is currently running:
https://gitlab.com/nbdkit/nbdkit/-/pipelines/491516164
I had previously fixed this failure in Alpine:
perl.c:45:10: fatal error: XSUB.h: No such file or directory
45 | #include <XSUB.h>
| ^~~~~~~~
This file comes from the perl-dev package which is missing on Alpine.
I had that fixed before (but only in the generated files) so I backed
that change out accidentally in the commit I just added to nbdkit.
While I remember, Martin had an alternate fix for this:
https://gitlab.com/nertpinx/nbdkit/-/commit/d93302150766415aaa7381c56120a...
but this fix is wrong - it completely disables Perl on every platform
(which is why it appears to work).
The true fix is this one (to libvirt-ci). I will send it upstream
separately.
diff --git a/guests/lcitool/lcitool/ansible/vars/mappings.yml
b/guests/lcitool/lcitool/ansible/vars/mappings.yml
index c05e59c..8b0825a 100644
--- a/guests/lcitool/lcitool/ansible/vars/mappings.yml
+++ b/guests/lcitool/lcitool/ansible/vars/mappings.yml
@@ -1617,6 +1617,8 @@ mappings:
deb: libperl-dev
rpm: perl-devel
pkg: perl5-devel
+ Alpine314: perl-dev
+ AlpineEdge: perl-dev
MacOS: perl5-devel
OpenSUSE:
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
6:57 a.m.
On Mon, Mar 14, 2022 at 11:54:57AM +0000, Richard W.M. Jones wrote:
On Mon, Mar 14, 2022 at 11:33:16AM +0000, Richard W.M. Jones wrote:
> Pipeline is currently running:
>
> https://gitlab.com/nbdkit/nbdkit/-/pipelines/491516164
I had previously fixed this failure in Alpine:
perl.c:45:10: fatal error: XSUB.h: No such file or directory
45 | #include <XSUB.h>
| ^~~~~~~~
This file comes from the perl-dev package which is missing on Alpine.
I had that fixed before (but only in the generated files) so I backed
that change out accidentally in the commit I just added to nbdkit.
While I remember, Martin had an alternate fix for this:
https://gitlab.com/nertpinx/nbdkit/-/commit/d93302150766415aaa7381c56120a...
but this fix is wrong - it completely disables Perl on every platform
(which is why it appears to work).
The true fix is this one (to libvirt-ci). I will send it upstream
separately.
diff --git a/guests/lcitool/lcitool/ansible/vars/mappings.yml
b/guests/lcitool/lcitool/ansible/vars/mappings.yml
index c05e59c..8b0825a 100644
--- a/guests/lcitool/lcitool/ansible/vars/mappings.yml
+++ b/guests/lcitool/lcitool/ansible/vars/mappings.yml
@@ -1617,6 +1617,8 @@ mappings:
deb: libperl-dev
rpm: perl-devel
pkg: perl5-devel
+ Alpine314: perl-dev
+ AlpineEdge: perl-dev
You can also set if for all Alpine versions with Alpine: perl-dev or all
distros using apk package manager with apk: perl-dev.
MacOS: perl5-devel
OpenSUSE:
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
7:02 a.m.
On Mon, Mar 14, 2022 at 12:57:13PM +0100, Martin Kletzander wrote:
You can also set if for all Alpine versions with Alpine: perl-dev or
all
distros using apk package manager with apk: perl-dev.
I was wondering about that :-)
How about this change?
https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/238
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
6:23 a.m.
On Mon, Mar 14, 2022 at 10:41:05AM +0000, Richard W.M. Jones wrote:
(b) lcitool wants to remove my perl conditionalization changes:
@@ -312,11 +309,10 @@ mingw32-fedora-35:
- mingw32-fedora-35-container
allow_failure: false
variables:
- NAME: fedora-35
CROSS: mingw32
- PERL: skip
- RUST: skip
GOLANG: skip
+ NAME: fedora-35
+ RUST: skip
which I added here:
https://gitlab.com/nbdkit/nbdkit/-/commit/414ee371ee2b1a12c6d45466d041322...
I'm not clear where I should have added those. The rust
conditionalization stuff doesn't seem to be part of lcitool so I guess
it's generated from another file somewhere.
Fixed this by modifying ci/manifest.yml.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
984
days inactive
986
days old
13 comments
3 participants
participants (3)
-
Daniel P. Berrangé -
Martin Kletzander -
Richard W.M. Jones