Notes on building libguestfs in a systemd-nspawn container
Last night I was tinkering with `systemd-nspawn` -- namespace based container for testing, I thought I'll post what I tried with libguestfs here: Prerequisite ------------ Because of an audit subsystem incompatibility bug - rhbz#966807[1], turn off auditing by booting the host w/ 'audit=0' on Kernel command line. (NOTE: There's work in progress[2] in upstream Kernel to fix this.) Create container ---------------- On the host, specify an installroot (/srv) and install a minimal Fedora 20 distribution: $ yum -y --releasever=20 --nogpg \ --installroot=/srv/testcontainer \ --disablerepo='*' --enablerepo=fedora \ install systemd passwd yum \ fedora-release vim-minimal Boot into the container, set a password: $ systemd-nspawn -D /srv/testcontainer [. . .] -bash-4.2# passwd Start the container w/ systemd: $ systemd-nspawn -bD /srv/testcontainer [. . .] -bash-4.2# Building Libguestfs ------------------- Inside the minimal Fedora 20 container, install libguestfs dependencies, clone the libguestfs git repository: -bash-4.2# yum-builddep libguestfs -y -bash-4.2# git clone git://github.com/libguestfs/libguestfs.git Build and test libguestfs in the container: -bash-4.2# cd libguestfs -bash-4.2# ./autogen.sh && time make 2>&1 \ | tee /tmp/libguestfs-compile.log -bash-4.2# time make -k check \ LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 2>&1 \ | tee /tmp/libguestfs-test.log Notes ----- - If you need to build a container without networking (once all the relevant dependencies are cloned/and network related operations are done), the container can be invoked with no network devices (except loopback): $ systemd-nspawn -bD /srv/testcontainer --private-network [. . .] -bash-4.2# - Single `make` job timing to compile everything: real 31m9.792s user 17m18.359s sys 13m17.868s - `make -k check` is still running as I write this, albeit a bit slow. [1] https://bugzilla.redhat.com/show_bug.cgi?id=966807 [2] https://www.redhat.com/archives/linux-audit/2013-May/msg00065.html -- /kashyap
On 01/30/2014 07:41 AM, Kashyap Chamarthy wrote:
Last night I was tinkering with `systemd-nspawn` -- namespace based container for testing, I thought I'll post what I tried with libguestfs here:
Prerequisite ------------
Because of an audit subsystem incompatibility bug - rhbz#966807[1], turn off auditing by booting the host w/ 'audit=0' on Kernel command line. (NOTE: There's work in progress[2] in upstream Kernel to fix this.)
Create container ----------------
On the host, specify an installroot (/srv) and install a minimal Fedora 20 distribution:
$ yum -y --releasever=20 --nogpg \ --installroot=/srv/testcontainer \ --disablerepo='*' --enablerepo=fedora \ install systemd passwd yum \ fedora-release vim-minimal
Boot into the container, set a password:
$ systemd-nspawn -D /srv/testcontainer [. . .] -bash-4.2# passwd
Start the container w/ systemd:
$ systemd-nspawn -bD /srv/testcontainer [. . .] -bash-4.2#
Building Libguestfs -------------------
Inside the minimal Fedora 20 container, install libguestfs dependencies, clone the libguestfs git repository:
-bash-4.2# yum-builddep libguestfs -y
-bash-4.2# git clone git://github.com/libguestfs/libguestfs.git
Build and test libguestfs in the container:
-bash-4.2# cd libguestfs
-bash-4.2# ./autogen.sh && time make 2>&1 \ | tee /tmp/libguestfs-compile.log
-bash-4.2# time make -k check \ LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 2>&1 \ | tee /tmp/libguestfs-test.log
Notes -----
- If you need to build a container without networking (once all the relevant dependencies are cloned/and network related operations are done), the container can be invoked with no network devices (except loopback):
$ systemd-nspawn -bD /srv/testcontainer --private-network [. . .] -bash-4.2#
- Single `make` job timing to compile everything:
real 31m9.792s user 17m18.359s sys 13m17.868s
For comparison, on the _host_, the same single `make` job timing: real 13m41.440s user 13m5.816s sys 1m9.911s
- `make -k check` is still running as I write this, albeit a bit slow.
This just finished (in the container): [. . .] grep -v -E '^(examples|gnulib|perl/(blib|examples)|po-docs|tests)/' | \ grep -v -E '/((guestfs|rc)_protocol\.c)$' | \ LC_ALL=C sort > po/POTFILES cd .; \ find builder mllib resize sparsify sysprep -name '*.ml' | \ LC_ALL=C sort > po/POTFILES-ml make[1]: Leaving directory `/root/libguestfs' make: *** [check-recursive] Error 1 GEN public-submodule-commit make: Target `check' not remade because of errors. real 474m53.630s user 325m54.254s sys 205m58.032s -bash-4.2# git log | head -1 commit c841d08d7084db69e81614d54423686cf0566ad6 Again, for comparison, `make -k check` on _host_: real 63m1.078s user 54m39.393s sys 12m8.130s I think it's worth noting that this is all with Btrfs on the host. And, the systemd[*] wiki page indeed warns: "YMMV if you use LVM or btrfs." It needs to be examined what the result would be if I repeat this on ext4 (and SSD). [*] http://www.freedesktop.org/wiki/Software/systemd/VirtualizedTesting/
[1] https://bugzilla.redhat.com/show_bug.cgi?id=966807 [2] https://www.redhat.com/archives/linux-audit/2013-May/msg00065.html
-- /kashyap
On Thu, Jan 30, 2014 at 11:50:35AM +0530, Kashyap Chamarthy wrote:
- Single `make` job timing to compile everything:
real 31m9.792s user 17m18.359s sys 13m17.868s
For comparison, on the _host_, the same single `make` job timing:
real 13m41.440s user 13m5.816s sys 1m9.911s
This is timing the build only? I'm surprised it is slower in the container. Is memory or # CPUs limited?
- `make -k check` is still running as I write this, albeit a bit slow.
This just finished (in the container):
[. . .] grep -v -E '^(examples|gnulib|perl/(blib|examples)|po-docs|tests)/' | \ grep -v -E '/((guestfs|rc)_protocol\.c)$' | \ LC_ALL=C sort > po/POTFILES cd .; \ find builder mllib resize sparsify sysprep -name '*.ml' | \ LC_ALL=C sort > po/POTFILES-ml make[1]: Leaving directory `/root/libguestfs' make: *** [check-recursive] Error 1 GEN public-submodule-commit make: Target `check' not remade because of errors.
real 474m53.630s user 325m54.254s sys 205m58.032s
-bash-4.2# git log | head -1 commit c841d08d7084db69e81614d54423686cf0566ad6
Again, for comparison, `make -k check` on _host_:
real 63m1.078s user 54m39.393s sys 12m8.130s
Is KVM available in the container? I've never tried that actually .. I suppose the next step is to make LIBGUESTFS_BACKEND=libvirt:lxc:/// work! Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
On Thu, Jan 30, 2014 at 10:28:31AM +0000, Richard W.M. Jones wrote:
On Thu, Jan 30, 2014 at 11:50:35AM +0530, Kashyap Chamarthy wrote:
Again, for comparison, `make -k check` on _host_:
real 63m1.078s user 54m39.393s sys 12m8.130s
Is KVM available in the container? I've never tried that actually ..
I don't believe so. IIRC systemd-nspawn will populate a tmpfs on /dev with only zero, null, full, random, unrandom, tty nodes. If you use libvirt to setup the container then you can tell libvirt to add in the /dev/kvm node too. This shows how to setup the basic Fedora OS container https://www.berrange.com/posts/2013/08/12/running-a-full-fedora-os-inside-a-... Then update the XML to passthrough the /dev/kvm char device as a hostdev: http://libvirt.org/formatdomain.html#elementsHostDevCaps Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 01/30/2014 03:58 PM, Richard W.M. Jones wrote:
On Thu, Jan 30, 2014 at 11:50:35AM +0530, Kashyap Chamarthy wrote:
- Single `make` job timing to compile everything:
real 31m9.792s user 17m18.359s sys 13m17.868s
For comparison, on the _host_, the same single `make` job timing:
real 13m41.440s user 13m5.816s sys 1m9.911s
This is timing the build only?
Yes. (I wonder if Btrfs matters here.)
I'm surprised it is slower in the container. Is memory or # CPUs limited?
I haven't done cgroups tuning or deep introspection or any such. It's just a default invocation of `systemd-nspawn`. That said, from inside the container: ========= -bash-4.2# cat /proc/cpuinfo | grep processor | wc -l 48 -bash-4.2# free -m total used free shared buffers cached Mem: 64259 12461 51798 49 1 10757 -/+ buffers/cache: 1703 62556 Swap: 13996 0 13996 -bash-4.2# ========= The machine is Intel(R) Xeon(R) CPU E7- 4807 @ 1.87GHz, it has 48 processors and 60 GB memory
- `make -k check` is still running as I write this, albeit a bit slow.
This just finished (in the container):
[. . .] grep -v -E '^(examples|gnulib|perl/(blib|examples)|po-docs|tests)/' | \ grep -v -E '/((guestfs|rc)_protocol\.c)$' | \ LC_ALL=C sort > po/POTFILES cd .; \ find builder mllib resize sparsify sysprep -name '*.ml' | \ LC_ALL=C sort > po/POTFILES-ml make[1]: Leaving directory `/root/libguestfs' make: *** [check-recursive] Error 1 GEN public-submodule-commit make: Target `check' not remade because of errors.
real 474m53.630s user 325m54.254s sys 205m58.032s
-bash-4.2# git log | head -1 commit c841d08d7084db69e81614d54423686cf0566ad6
Again, for comparison, `make -k check` on _host_:
real 63m1.078s user 54m39.393s sys 12m8.130s
Is KVM available in the container? I've never tried that actually ..
No it isn't (as Dan noted in his next thread) ========= -bash-4.2# file /dev/kvm /dev/kvm: ERROR: cannot open `/dev/kvm' (No such file or directory) ========= -bash-4.2# virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking for device /dev/kvm : FAIL (Check that the 'kvm-intel' or 'kvm-amd' modules are loaded & the BIOS has enabled virtualization) QEMU: Checking for device /dev/vhost-net : WARN (Load the 'vhost_net' module to improve performance of virtio networking) QEMU: Checking for device /dev/net/tun : FAIL (Load the 'tun' module to enable networking for QEMU guests) LXC: Checking for Linux >= 2.6.26 : PASS ========= Despite reading from the `systemd-nspawn` man page: ". . .kernel modules may not be loaded from within the container." I purposefully tried from inside the container: ========= -bash-4.2# modprobe kvm_intel -bash-4.2# echo $? 1 -bash-4.2# file /dev/kvm /dev/kvm: ERROR: cannot open `/dev/kvm' (No such file or directory) -bash-4.2# =========
I suppose the next step is to make LIBGUESTFS_BACKEND=libvirt:lxc:/// work!
Rich.
-- /kashyap
On Thu, Jan 30, 2014 at 04:34:04PM +0530, Kashyap Chamarthy wrote:
On 01/30/2014 03:58 PM, Richard W.M. Jones wrote:
- `make -k check` is still running as I write this, albeit a bit slow.
This just finished (in the container):
[. . .] grep -v -E '^(examples|gnulib|perl/(blib|examples)|po-docs|tests)/' | \ grep -v -E '/((guestfs|rc)_protocol\.c)$' | \ LC_ALL=C sort > po/POTFILES cd .; \ find builder mllib resize sparsify sysprep -name '*.ml' | \ LC_ALL=C sort > po/POTFILES-ml make[1]: Leaving directory `/root/libguestfs' make: *** [check-recursive] Error 1 GEN public-submodule-commit make: Target `check' not remade because of errors.
real 474m53.630s user 325m54.254s sys 205m58.032s
-bash-4.2# git log | head -1 commit c841d08d7084db69e81614d54423686cf0566ad6
Again, for comparison, `make -k check` on _host_:
real 63m1.078s user 54m39.393s sys 12m8.130s
Is KVM available in the container? I've never tried that actually ..
No it isn't (as Dan noted in his next thread)
========= -bash-4.2# file /dev/kvm /dev/kvm: ERROR: cannot open `/dev/kvm' (No such file or directory) ========= -bash-4.2# virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking for device /dev/kvm : FAIL (Check that the 'kvm-intel' or 'kvm-amd' modules are loaded & the BIOS has enabled virtualization) QEMU: Checking for device /dev/vhost-net : WARN (Load the 'vhost_net' module to improve performance of virtio networking) QEMU: Checking for device /dev/net/tun : FAIL (Load the 'tun' module to enable networking for QEMU guests) LXC: Checking for Linux >= 2.6.26 : PASS =========
Despite reading from the `systemd-nspawn` man page:
". . .kernel modules may not be loaded from within the container."
I purposefully tried from inside the container:
With container based virt there is only one kernel image, so any modules you want must be loaded in the host. Libvirt "passthrough" of char/block devices simply involves libvirt doing mknod in the /dev tmpfs it sets up. The container itself is blocked from doing any 'mknod' calls since that'd be a security risk. Hence you must list any desired device nodes in the XML config. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 01/30/2014 04:38 PM, Daniel P. Berrange wrote: [. . .]
Despite reading from the `systemd-nspawn` man page:
". . .kernel modules may not be loaded from within the container."
I purposefully tried from inside the container:
With container based virt there is only one kernel image,
Noted, that's one of the main aspects, right, of containers: single Kernel (also a single point of attack-surface; no custom Kernels, etc)[1] But I see the use-case of systemd-nspawn: quick development/debugging just like chroot, but better.
so any modules you want must be loaded in the host. Libvirt "passthrough" of char/block devices simply involves libvirt doing mknod in the /dev tmpfs it sets up. The container itself is blocked from doing any 'mknod' calls since that'd be a security risk. Hence you must list any desired device nodes in the XML config.
Thanks for the explanation. I have to try libvirt-lxc tools next. Also on my todo-list to try: $ virt-sandbox mock [Build a package] I see that the above provides a default SELinux 'seclabel' element. Have to test yet. Meanwhile, I stumbled across an upstream thread[2][3] of yours this morning & learnt re: a regression with user namespaces containers [1] http://rwmj.wordpress.com/2013/06/19/the-boring-truth-full-virtualization-an... [2] https://lists.linuxfoundation.org/pipermail/containers/2013-November/033635.... [3] https://bugzilla.redhat.com/show_bug.cgi?id=917708 -- /kashyap
On Thu, Jan 30, 2014 at 05:07:23PM +0530, Kashyap Chamarthy wrote:
On 01/30/2014 04:38 PM, Daniel P. Berrange wrote:
[. . .]
Despite reading from the `systemd-nspawn` man page:
". . .kernel modules may not be loaded from within the container."
I purposefully tried from inside the container:
With container based virt there is only one kernel image,
Noted, that's one of the main aspects, right, of containers: single Kernel (also a single point of attack-surface; no custom Kernels, etc)[1]
But I see the use-case of systemd-nspawn: quick development/debugging just like chroot, but better.
so any modules you want must be loaded in the host. Libvirt "passthrough" of char/block devices simply involves libvirt doing mknod in the /dev tmpfs it sets up. The container itself is blocked from doing any 'mknod' calls since that'd be a security risk. Hence you must list any desired device nodes in the XML config.
Thanks for the explanation. I have to try libvirt-lxc tools next. Also on my todo-list to try:
$ virt-sandbox mock
[Build a package]
I see that the above provides a default SELinux 'seclabel' element. Have to test yet.
Meanwhile, I stumbled across an upstream thread[2][3] of yours this morning & learnt re: a regression with user namespaces containers
Nb user namespaces aren't relevant here. Nothing you're using / trying here involves user namespaces at all. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 01/30/2014 05:10 PM, Daniel P. Berrange wrote:
On Thu, Jan 30, 2014 at 05:07:23PM +0530, Kashyap Chamarthy wrote:
On 01/30/2014 04:38 PM, Daniel P. Berrange wrote:
[. . .]
Despite reading from the `systemd-nspawn` man page:
". . .kernel modules may not be loaded from within the container."
I purposefully tried from inside the container:
With container based virt there is only one kernel image,
Noted, that's one of the main aspects, right, of containers: single Kernel (also a single point of attack-surface; no custom Kernels, etc)[1]
But I see the use-case of systemd-nspawn: quick development/debugging just like chroot, but better.
so any modules you want must be loaded in the host. Libvirt "passthrough" of char/block devices simply involves libvirt doing mknod in the /dev tmpfs it sets up. The container itself is blocked from doing any 'mknod' calls since that'd be a security risk. Hence you must list any desired device nodes in the XML config.
Thanks for the explanation. I have to try libvirt-lxc tools next. Also on my todo-list to try:
$ virt-sandbox mock
[Build a package]
I see that the above provides a default SELinux 'seclabel' element. Have to test yet.
Meanwhile, I stumbled across an upstream thread[2][3] of yours this morning & learnt re: a regression with user namespaces containers
Nb user namespaces aren't relevant here. Nothing you're using / trying here involves user namespaces at all.
Sorry, didn't mean to imply they both are connected (it was my poor wording). I came across it while I was learning about user namespaces, its current state in Fedora. -- /kashyap
participants (3)
-
Daniel P. Berrange -
Kashyap Chamarthy -
Richard W.M. Jones