Adventures in building libguestfs on non-x86 architectures for Debian
Hi, things are progressing slowly, but I feel that eventually I'll be getting there: supermin can be built on all Linux-based architectures that are part of Debian/unstable[1] -- (except for sparc, but I'll ignore that for now.) libguestfs on the other hand currently fails launching the appliance (necessary for running tests) on everything but x86 and mips. Apparently, qemu-system-arm 2.0 as currently available through Debian/unstable doesn't want to tell us anything about available devices unless we specify a machine type. The patch below works around that issue, but I haven't been able to run qemu-system-arm with a supermin appliance on Debian's armhf porterbox so far: 100% CPU usage, no output from the kernel. For armel, there's no kernel with device tree blobs, so I think that I'll need to use linux-image-versatile there or ignore armel altogether. powerpc: qemu-system-ppc does not know about the "pseries" machine, I have figured out that I need to use qemu-system-ppc64 for that. So far, I haven't been able to manually boot a Supermin appliance using qemu-system-ppc, either, though. s390x does not seem to know about virtio-blk-pci. Cheers, -Hilko [1] https://buildd.debian.org/status/package.php?p=supermin [2] https://buildd.debian.org/status/package.php?p=libguestfs diff --git a/src/launch-direct.c b/src/launch-direct.c index 1460c56..2332368 100644 --- a/src/launch-direct.c +++ b/src/launch-direct.c @@ -1020,6 +1020,10 @@ test_qemu (guestfs_h *g, struct backend_direct_data *data) guestfs___cmd_add_arg (cmd3, g->hv); guestfs___cmd_add_arg (cmd3, "-display"); guestfs___cmd_add_arg (cmd3, "none"); +#ifdef MACHINE_TYPE + guestfs___cmd_add_arg (cmd3, "-M"); + guestfs___cmd_add_arg (cmd3, MACHINE_TYPE); +#endif guestfs___cmd_add_arg (cmd3, "-machine"); guestfs___cmd_add_arg (cmd3, "accel=kvm:tcg"); guestfs___cmd_add_arg (cmd3, "-device");
On Tue, Apr 29, 2014 at 11:18:01PM +0200, Hilko Bengen wrote:
Hi,
things are progressing slowly, but I feel that eventually I'll be getting there: supermin can be built on all Linux-based architectures that are part of Debian/unstable[1] -- (except for sparc, but I'll ignore that for now.) libguestfs on the other hand currently fails launching the appliance (necessary for running tests) on everything but x86 and mips.
Apparently, qemu-system-arm 2.0 as currently available through Debian/unstable doesn't want to tell us anything about available devices unless we specify a machine type. The patch below works around that issue, but I haven't been able to run qemu-system-arm with a supermin appliance on Debian's armhf porterbox so far: 100% CPU usage, no output from the kernel.
ARM is always fun to debug, but 32 bit ARM should be working. It works for me anyway (using Fedora and qemu 1.x). The way to debug your spin is to use gdb (the following assumes you have access to the hardware; if you don't, my recommendation is to buy a Cubietruck!). Uncomment the following lines of code: https://github.com/libguestfs/libguestfs/blob/master/src/launch-direct.c#L40... When libguestfs runs, it will pause when the appliance starts. Attach gdb to the appliance by following the instructions there, and 'cont' it, then when it spins hit ^C and get a stack trace to see where in the kernel it got stuck. You can also dump the kernel log (log_buf) to see if there are any lost kernel messages. We should have aarch64 working in a matter of weeks. The problem is keeping it working, since we have no CI-style tests on these architectures. Anything from kernel updates to qemu updates to libguestfs changes can break it, and we wouldn't notice for a while.
For armel, there's no kernel with device tree blobs, so I think that I'll need to use linux-image-versatile there or ignore armel altogether.
powerpc: qemu-system-ppc does not know about the "pseries" machine, I have figured out that I need to use qemu-system-ppc64 for that. So far, I haven't been able to manually boot a Supermin appliance using qemu-system-ppc, either, though.
ppc64 ought to work. At least, it WFM (on Fedora ...). ppc64le is a different matter. So far I've not been able to get access to any POWER8 hardware. However Michel Normand from IBM has looked at getting the virt stack working on ppc64le on Fedora. I haven't tried ppc 32 bit for a long time, and don't particularly care about it. AFAIK the hardware is obsolete.
s390x does not seem to know about virtio-blk-pci.
s390x is problematic in a few ways: - No OCaml native compiler - KVM exists but is completely unknown territory. I have access to real s390x hardware but I've not spent any serious time trying to get libguestfs to work.
[1] https://buildd.debian.org/status/package.php?p=supermin [2] https://buildd.debian.org/status/package.php?p=libguestfs
diff --git a/src/launch-direct.c b/src/launch-direct.c index 1460c56..2332368 100644 --- a/src/launch-direct.c +++ b/src/launch-direct.c @@ -1020,6 +1020,10 @@ test_qemu (guestfs_h *g, struct backend_direct_data *data) guestfs___cmd_add_arg (cmd3, g->hv); guestfs___cmd_add_arg (cmd3, "-display"); guestfs___cmd_add_arg (cmd3, "none"); +#ifdef MACHINE_TYPE + guestfs___cmd_add_arg (cmd3, "-M"); + guestfs___cmd_add_arg (cmd3, MACHINE_TYPE); +#endif guestfs___cmd_add_arg (cmd3, "-machine"); guestfs___cmd_add_arg (cmd3, "accel=kvm:tcg"); guestfs___cmd_add_arg (cmd3, "-device");
ACK. Thanks for looking into this! Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
* Richard W.M. Jones:
Apparently, qemu-system-arm 2.0 as currently available through Debian/unstable doesn't want to tell us anything about available devices unless we specify a machine type. The patch below works around that issue, but I haven't been able to run qemu-system-arm with a supermin appliance on Debian's armhf porterbox so far: 100% CPU usage, no output from the kernel.
ARM is always fun to debug, but 32 bit ARM should be working. It works for me anyway (using Fedora and qemu 1.x).
I just got around to revisiting the problem. Using qemu-system-arm_2.0.0+dfsg-6 from Debian/unstable, booting vexpress-a15 doesn't seem to work, but vexpress-a9 does. I successfully built and did a "make quickcheck" on an armhf porterbox with the patch attached below. Is there any specific reason why -a15 was chosen that might make this a bad idea? I alos noticed that there seems to be a plan to use -M virt at some point for 32bit arm. Can qemu generate the device tree internally yet? Cheers, -Hilko
On Sun, Jul 06, 2014 at 09:43:59PM +0200, Hilko Bengen wrote:
* Richard W.M. Jones:
Apparently, qemu-system-arm 2.0 as currently available through Debian/unstable doesn't want to tell us anything about available devices unless we specify a machine type. The patch below works around that issue, but I haven't been able to run qemu-system-arm with a supermin appliance on Debian's armhf porterbox so far: 100% CPU usage, no output from the kernel.
ARM is always fun to debug, but 32 bit ARM should be working. It works for me anyway (using Fedora and qemu 1.x).
I just got around to revisiting the problem. Using qemu-system-arm_2.0.0+dfsg-6 from Debian/unstable, booting vexpress-a15 doesn't seem to work, but vexpress-a9 does. I successfully built and did a "make quickcheck" on an armhf porterbox with the patch attached below. Is there any specific reason why -a15 was chosen that might make this a bad idea?
QEMU can emulate any of Cortex-A8, A9 and A15. However we don't specify the -cpu explicitly, which we probably should do. I wonder if the default is different on Fedora and Debian for some reason? Of those only A15 is capable of virtualization, which for TCG only matters if you want to do nested virt (and is irrelevant for libguestfs). In the hardware-assisted case (KVM), qemu-system-arm only emulates A15, which is necessary because only A7 and A15 can do virtualization and so you would only be running qemu-system-arm in KVM mode on A7 or A15. (In fact when I did get KVM working on my A7 system, I had to patch qemu for exactly this reason). So I think it's better to use A15, if it works. Can you see if adding `-cpu cortex-a15' works instead?
I alos noticed that there seems to be a plan to use -M virt at some point for 32bit arm. Can qemu generate the device tree internally yet?
-M virt is definitely preferable. For Aarch64 we're using -M virt. Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
participants (2)
-
Hilko Bengen -
Richard W.M. Jones