Issue with virt-builder on Ubuntu 14.04
Hello, I am running into problems with virt-builder on my Ubuntu-14.04 install. Please find the checklist below: 1. What are you trying to achieve? I'm trying to build a standard Ubuntu 14.04 disk image. 2. What exact commands did you run? I ran "virt-builder ubuntu-14.04" 3. What was the precise error/output of these commands? The process gets stuck indefinitely at "Opening the new disk." I even left it over night and it did not get any farther. 4. Enable debugging, run the commands again, and capture the complete output. The file attached contains the output from the command with debugging enabled. 5. Versions of installed software: libguestfs: 1.28.10 compiled from source tarball but not installed supermin: 5.1.13 compiled from source tarball but not installed, used instructions from https://rwmj.wordpress.com/2014/03/08/tip-old-supermin-new-libguestfs-and-v-... to compile libguestfs with supermin5 operating system: Ubuntu 14.04.2 x64 kernel: 3.13.0-55-generic #94-Ubuntu SMP Thu Jun 18 00:27:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Additional information: This is a new issue, I used to use libguestfs 1.28.6 (compiled from source) and had no issues in the past until this weekend. So I compiled the newest version (1.28.10) and still have this issue. Additionally, I tried compiling and running it on a virtual machine (Ubuntu 14.04 image created months ago with virt-builder when it was working) and that works as expected. Therefore I believe that it is an issue with something on my host, but I can't seem to narrow it down. I can still use other libguestfs tools such as virt-df, and virt-inspector (those are the only two that I've tried) Any help is appreciated. Thanks, Shawn
On Mon, Jun 29, 2015 at 08:49:26PM -0600, Shawn Greig wrote: [...] Thanks for the detailed bug report. The issue is indeed something strange with your host or KVM or your host kernel (3.13.0-55-generic). As you can see from the end of the messages:
[ 0.371018] EXT4-fs (sdb): mounting ext2 file system using the ext4 subsystem [ 0.393840] EXT4-fs (sdb): mounted filesystem without journal. Opts: supermin: chroot Starting /init script ... ln: failed to create symbolic link '/etc/mtab': File exists /init: 55: /init: systemd-tmpfiles: not found [ 1.687079] systemd-udevd[91]: starting version 204 [ 1.808148] random: nonblocking pool is initialized [ 1.886029] ACPI Exception: AE_BAD_PARAMETER, Thread 487102464 could not acquire Mutex [0x1] (20131115/utmutex-285) [ 1.886591] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 /init: 79: /init: cannot create /sys/block/{h,s,ub,v}d*/queue/scheduler: Directory nonexistent /sbin/dhclient-script: line 31: /etc/fstab: No such file or directory [ 2.935872] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
... the kernel boots quite far, and then hangs in the init script (see appliance/init). Some things you can try: - Force qemu to use TCG (no KVM) mode to see if that makes a difference: export LIBGUESTFS_BACKEND_SETTINGS=force_tcg - Use 'libguestfs-test-tool' for testing to see if that makes a difference. In the build directory, run: make quickcheck - Try a different appliance kernel by setting these variables, but choose different kernel versions by looking in your /boot directory first: export SUPERMIN_KERNEL_VERSION=3.13.0-55-generic export SUPERMIN_KERNEL=/boot/vmlinuz-$SUPERMIN_KERNEL_VERSION export SUPERMIN_MODULES=/lib/modules/$SUPERMIN_KERNEL_VERSION rm -rf tmp/.guestfs-* make quickcheck - Add debugging 'set -x' to the top of appliance/init in the source tree to see if it fails while running a specific command. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
- I tried forcing TCG and it gets to the same spot and stops. - running make quickcheck passes the tests (see attached) - I tried a different kernel (3.13.0-49-generic) and again it passes the quickcheck but virt-builder hangs. - I added 'set -x' to the top of the init script and I don't see any additional output (see attached) Shawn On Tue, Jun 30, 2015 at 2:06 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Mon, Jun 29, 2015 at 08:49:26PM -0600, Shawn Greig wrote: [...]
Thanks for the detailed bug report.
The issue is indeed something strange with your host or KVM or your host kernel (3.13.0-55-generic). As you can see from the end of the messages:
[ 0.371018] EXT4-fs (sdb): mounting ext2 file system using the ext4 subsystem [ 0.393840] EXT4-fs (sdb): mounted filesystem without journal. Opts: supermin: chroot Starting /init script ... ln: failed to create symbolic link '/etc/mtab': File exists /init: 55: /init: systemd-tmpfiles: not found [ 1.687079] systemd-udevd[91]: starting version 204 [ 1.808148] random: nonblocking pool is initialized [ 1.886029] ACPI Exception: AE_BAD_PARAMETER, Thread 487102464 could not acquire Mutex [0x1] (20131115/utmutex-285) [ 1.886591] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 /init: 79: /init: cannot create /sys/block/{h,s,ub,v}d*/queue/scheduler: Directory nonexistent /sbin/dhclient-script: line 31: /etc/fstab: No such file or directory [ 2.935872] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
... the kernel boots quite far, and then hangs in the init script (see appliance/init).
Some things you can try:
- Force qemu to use TCG (no KVM) mode to see if that makes a difference:
export LIBGUESTFS_BACKEND_SETTINGS=force_tcg
- Use 'libguestfs-test-tool' for testing to see if that makes a difference. In the build directory, run:
make quickcheck
- Try a different appliance kernel by setting these variables, but choose different kernel versions by looking in your /boot directory first:
export SUPERMIN_KERNEL_VERSION=3.13.0-55-generic export SUPERMIN_KERNEL=/boot/vmlinuz-$SUPERMIN_KERNEL_VERSION export SUPERMIN_MODULES=/lib/modules/$SUPERMIN_KERNEL_VERSION rm -rf tmp/.guestfs-* make quickcheck
- Add debugging 'set -x' to the top of appliance/init in the source tree to see if it fails while running a specific command.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Tue, Jun 30, 2015 at 09:07:25AM -0600, Shawn Greig wrote:
- I tried forcing TCG and it gets to the same spot and stops.
That excludes the possibility that it's a KVM or kernel bug (probably).
- running make quickcheck passes the tests (see attached)
- I tried a different kernel (3.13.0-49-generic) and again it passes the quickcheck but virt-builder hangs.
Add 'virt-builder -v -x' flags and see where it's hanging.
- I added 'set -x' to the top of the init script and I don't see any additional output (see attached)
Somehow this didn't have any effect. You probably didn't do a 'make' before the 'make quickcheck'. It would still be useful to see the 'set -x' output to see if a particular command is hanging. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
Rich you're right, I forgot to run make. I added 'set -x' and reran make and then ran 'virt-builder -v -x' and it seems to hang at dhclient (see attached log). Shawn On Tue, Jun 30, 2015 at 9:17 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Tue, Jun 30, 2015 at 09:07:25AM -0600, Shawn Greig wrote:
- I tried forcing TCG and it gets to the same spot and stops.
That excludes the possibility that it's a KVM or kernel bug (probably).
- running make quickcheck passes the tests (see attached)
- I tried a different kernel (3.13.0-49-generic) and again it passes the quickcheck but virt-builder hangs.
Add 'virt-builder -v -x' flags and see where it's hanging.
- I added 'set -x' to the top of the init script and I don't see any additional output (see attached)
Somehow this didn't have any effect. You probably didn't do a 'make' before the 'make quickcheck'. It would still be useful to see the 'set -x' output to see if a particular command is hanging.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Tue, Jun 30, 2015 at 09:41:45AM -0600, Shawn Greig wrote:
Rich you're right, I forgot to run make. I added 'set -x' and reran make and then ran 'virt-builder -v -x' and it seems to hang at dhclient (see attached log).
+ dhclient --version + dhclient /sbin/dhclient-script: line 31: /etc/fstab: No such file or directory
Yup, dhclient has been an ongoing problem. Most recently: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775514 https://bugzilla.redhat.com/show_bug.cgi?id=1224795 It may help to run dhclient in the appliance under strace to see where exactly it has stopped. The appliance should contain strace (check for it in appliance/packagelist.in and appliance/supermin.d/packages), and if it does you can just edit the appliance/init script like this: - dhclient + strace dhclient The other option is to use dhcpcd, but that also requires patching the init script. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
Yeah, it's definitely an issue with dhclient. I patched the init script to use dhcpcd instead and everything is working as expected now. I've attached a log containing the strace of dhclient if that helps you or someone else in the future. Thanks for all your help Rich. Shawn On Tue, Jun 30, 2015 at 9:53 AM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Tue, Jun 30, 2015 at 09:41:45AM -0600, Shawn Greig wrote:
Rich you're right, I forgot to run make. I added 'set -x' and reran make and then ran 'virt-builder -v -x' and it seems to hang at dhclient (see attached log).
+ dhclient --version + dhclient /sbin/dhclient-script: line 31: /etc/fstab: No such file or directory
Yup, dhclient has been an ongoing problem. Most recently:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775514 https://bugzilla.redhat.com/show_bug.cgi?id=1224795
It may help to run dhclient in the appliance under strace to see where exactly it has stopped. The appliance should contain strace (check for it in appliance/packagelist.in and appliance/supermin.d/packages), and if it does you can just edit the appliance/init script like this:
- dhclient + strace dhclient
The other option is to use dhcpcd, but that also requires patching the init script.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Tue, Jun 30, 2015 at 11:50:57AM -0600, Shawn Greig wrote:
Yeah, it's definitely an issue with dhclient. I patched the init script to use dhcpcd instead and everything is working as expected now.
/sbin/dhclient-script: line 31: /etc/fstab: No such file or directory So it seems to actually hang when /etc/fstab isn't present. Depending on the base OS, the libguestfs appliance may or may not have /etc/fstab -- it's not really required. It's a bug in whatever provides dhclient-script if dhclient-script hangs when this file is not present. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
participants (2)
-
Richard W.M. Jones -
Shawn Greig