Xen drivers for virt-builder images
I've just been playing around with virt-builder, and it seems like a pretty useful tool. However, it seems that in a number of the images, the initrd file is missing drivers for Xen virtual block and network devices. This results in a VM that can't find its root device. The files needed are drivers/block/xen-blkfront.ko and drivers/net/xen-netfront.ko. They seem to be present in centos-6 and ubuntu-14.04, but not in rhel-7rc or fedora-20. Whom would I contact to get these added? Thanks! -George
On Wed, Jun 11, 2014 at 06:15:33PM +0100, George Dunlap wrote:
I've just been playing around with virt-builder, and it seems like a pretty useful tool. However, it seems that in a number of the images, the initrd file is missing drivers for Xen virtual block and network devices. This results in a VM that can't find its root device.
The files needed are drivers/block/xen-blkfront.ko and drivers/net/xen-netfront.ko. They seem to be present in centos-6 and ubuntu-14.04, but not in rhel-7rc or fedora-20.
Whom would I contact to get these added?
That would likely be me or Pino Toscano, but read on :-) Firstly I'd like to say that the current images shipped in http://libguestfs.org/download/builder/ are a temporary solution that has turned out to be more permanent than we wished for. Eventually we want to simply use the cloud images shipped by the Linux distros, and just provide simple metadata and the virt-builder front end. When we do that (and it's not a huge change) then the issue of drivers for Xen, VMware, virtio, etc will be up to the distros to solve. However at the moment we could add Xen drivers to our current scripts, located here: https://github.com/libguestfs/libguestfs/tree/master/builder/website (*.sh) For Fedora and RHEL, it would require a modest change to the kickstart embedded in those scripts, plus rebuilding the images and reuploading them. Do you know what extra packages have to be installed or commands need to be run to enable Xen drivers in Fedora? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Wed, Jun 11, 2014 at 8:34 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Jun 11, 2014 at 06:15:33PM +0100, George Dunlap wrote:
I've just been playing around with virt-builder, and it seems like a pretty useful tool. However, it seems that in a number of the images, the initrd file is missing drivers for Xen virtual block and network devices. This results in a VM that can't find its root device.
The files needed are drivers/block/xen-blkfront.ko and drivers/net/xen-netfront.ko. They seem to be present in centos-6 and ubuntu-14.04, but not in rhel-7rc or fedora-20.
Whom would I contact to get these added?
That would likely be me or Pino Toscano, but read on :-)
Firstly I'd like to say that the current images shipped in http://libguestfs.org/download/builder/ are a temporary solution that has turned out to be more permanent than we wished for. Eventually we want to simply use the cloud images shipped by the Linux distros, and just provide simple metadata and the virt-builder front end. When we do that (and it's not a huge change) then the issue of drivers for Xen, VMware, virtio, etc will be up to the distros to solve.
Right -- that makes sense. I'm helping out with the CentOS Virt SIG at the moment, so I can try to make that happen for CentOS when the time comes. :-)
However at the moment we could add Xen drivers to our current scripts, located here:
https://github.com/libguestfs/libguestfs/tree/master/builder/website (*.sh)
For Fedora and RHEL, it would require a modest change to the kickstart embedded in those scripts, plus rebuilding the images and reuploading them.
Do you know what extra packages have to be installed or commands need to be run to enable Xen drivers in Fedora?
The kernel already has a lot of the Xen stuff enabled. AFAICT from looking at the boot output it's just missing those two modules in the initrd. (Although it's possible you'll need more.) I'm pretty sure they should just come in the normal kernel rpm; there aren't really any user-space components necessary for basic functionality of a Xen guest. (There are a few tools which someone might want to install for one purpose or another -- tools for querying Xen state or acting as a "driver domain" -- but they're something that should be added as needed, not in the core.) Dario has more familiarity with Fedora and Xen -- Dario, any comments? Thanks for your help, -George
On Wed, Jun 11, 2014 at 10:02:22PM +0100, George Dunlap wrote:
On Wed, Jun 11, 2014 at 8:34 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
Do you know what extra packages have to be installed or commands need to be run to enable Xen drivers in Fedora?
The kernel already has a lot of the Xen stuff enabled. AFAICT from looking at the boot output it's just missing those two modules in the initrd. (Although it's possible you'll need more.) I'm pretty sure they should just come in the normal kernel rpm; there aren't really any user-space components necessary for basic functionality of a Xen guest. (There are a few tools which someone might want to install for one purpose or another -- tools for querying Xen state or acting as a "driver domain" -- but they're something that should be added as needed, not in the core.)
I suspect we may need to run a dracut command in %post. Something like: dracut --add-drivers "some list of xen kmods ..." I'm not sure which Xen drivers are required. The Fedora 21 kernel has 13 files named 'xen*.ko.xz'. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Hello everyone, On gio, 2014-06-12 at 08:20 +0100, Richard W.M. Jones wrote:
On Wed, Jun 11, 2014 at 10:02:22PM +0100, George Dunlap wrote:
On Wed, Jun 11, 2014 at 8:34 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
Do you know what extra packages have to be installed or commands need to be run to enable Xen drivers in Fedora?
The kernel already has a lot of the Xen stuff enabled. AFAICT from looking at the boot output it's just missing those two modules in the initrd. (Although it's possible you'll need more.) I'm pretty sure they should just come in the normal kernel rpm; there aren't really any user-space components necessary for basic functionality of a Xen guest. (There are a few tools which someone might want to install for one purpose or another -- tools for querying Xen state or acting as a "driver domain" -- but they're something that should be added as needed, not in the core.)
I suspect we may need to run a dracut command in %post. Something like:
dracut --add-drivers "some list of xen kmods ..."
I'm not sure which Xen drivers are required. The Fedora 21 kernel has 13 files named 'xen*.ko.xz'.
So, first of all, forgive my libguestfs / virt-builder ignorance. :-) About the modules, what I know is that if I point virt-install to this location: http://mirror.euserv.net/linux/fedora/linux/releases/20/Fedora/x86_64/os/ it just works, i.e., the installer is able to see the virtual disk and network interface while running in a Xen (PV) guest. Looking inside the initrd.img that the above uses, here's what I find: $ find lib/modules/3.11.10-301.fc20.x86_64/ -iname xen* lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/block/xen-blkfront.ko lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/net/xen-netback lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/net/xen-netback/xen-netback.ko lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/net/xen-netfront.ko So, if we're talking about an initrd, or an initrd-like thing, the above is probably all we need. Is that what we're talking about? :-) Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
On Thu, Jun 12, 2014 at 1:20 PM, Dario Faggioli <dario.faggioli@citrix.com> wrote:
Hello everyone,
On gio, 2014-06-12 at 08:20 +0100, Richard W.M. Jones wrote:
On Wed, Jun 11, 2014 at 10:02:22PM +0100, George Dunlap wrote:
On Wed, Jun 11, 2014 at 8:34 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
Do you know what extra packages have to be installed or commands need to be run to enable Xen drivers in Fedora?
The kernel already has a lot of the Xen stuff enabled. AFAICT from looking at the boot output it's just missing those two modules in the initrd. (Although it's possible you'll need more.) I'm pretty sure they should just come in the normal kernel rpm; there aren't really any user-space components necessary for basic functionality of a Xen guest. (There are a few tools which someone might want to install for one purpose or another -- tools for querying Xen state or acting as a "driver domain" -- but they're something that should be added as needed, not in the core.)
I suspect we may need to run a dracut command in %post. Something like:
dracut --add-drivers "some list of xen kmods ..."
I'm not sure which Xen drivers are required. The Fedora 21 kernel has 13 files named 'xen*.ko.xz'.
So, first of all, forgive my libguestfs / virt-builder ignorance. :-)
About the modules, what I know is that if I point virt-install to this location: http://mirror.euserv.net/linux/fedora/linux/releases/20/Fedora/x86_64/os/
it just works, i.e., the installer is able to see the virtual disk and network interface while running in a Xen (PV) guest.
Looking inside the initrd.img that the above uses, here's what I find:
$ find lib/modules/3.11.10-301.fc20.x86_64/ -iname xen* lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/block/xen-blkfront.ko lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/net/xen-netback lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/net/xen-netback/xen-netback.ko lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/net/xen-netfront.ko
I'm not sure why they have xen-netback in there -- that's a network *provider*, not a network *consumer*. It certainly shouldn't be necessary for booting! xen-blkfront and xen-netfront were the two that I mentioned in my original e-mail; I'm pretty sure just adding those two should work. For the KVM folks: unlike virtio, which is provided to the guest by qemu, the traditional Xen PV devices are provided by another VM on the system. This is typically domain 0, but can actually be any VM. The code providing the virtual device is called the "back-end", and the code consuming the virtual device is called the "front-end". All you need in the initrd are front-ends for disk and network -- xen-blkfront and xen-netfront. -George
On Thu, Jun 12, 2014 at 02:20:49PM +0200, Dario Faggioli wrote:
So, first of all, forgive my libguestfs / virt-builder ignorance. :-)
About the modules, what I know is that if I point virt-install to this location: http://mirror.euserv.net/linux/fedora/linux/releases/20/Fedora/x86_64/os/
it just works, i.e., the installer is able to see the virtual disk and network interface while running in a Xen (PV) guest.
Looking inside the initrd.img that the above uses, here's what I find:
$ find lib/modules/3.11.10-301.fc20.x86_64/ -iname xen* lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/block/xen-blkfront.ko lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/net/xen-netback lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/net/xen-netback/xen-netback.ko lib/modules/3.11.10-301.fc20.x86_64/kernel/drivers/net/xen-netfront.ko
So, if we're talking about an initrd, or an initrd-like thing, the above is probably all we need. Is that what we're talking about? :-)
I believe the problem is that we use KVM to build the virt-builder guest images. When the kernel package is installed by kickstart, since the hypervisor at build time != Xen, dracut decides not to include the Xen drivers. I think we can fix this by running dracut again in %post, and I propose that running dracut as follows should fix this: %post dracut --add-drivers "xen:vbd xen:vif" (assuming what Olaf says works), or: %post dracut --add-drivers "xen-blkfront xen-netback xen-netfront" I will try it shortly. Rich. PS. Why would a Xen domU need the netback driver? -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
Not quite as easy as I imagined. However by using the attached patch I have managed to add the two Xen drivers to the initramfs. (I haven't been able to test that it boots in Xen however) I'm just building an updated Fedora 20 image which I'll upload later today. Are there other guests that need to be updated in the same way? The RHEL 7 RC guest probably no longer works, since it looks like we've got and deleted the binary packages from ftp.redhat.com. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Thu, Jun 12, 2014 at 03:12:46PM +0100, Richard W.M. Jones wrote:
Not quite as easy as I imagined. However by using the attached patch I have managed to add the two Xen drivers to the initramfs. (I haven't been able to test that it boots in Xen however)
I'm just building an updated Fedora 20 image which I'll upload later today.
The F20 image has been uploaded. You should notice when you install it that it will download the full disk image again, and also that a file "fedora-20.x86_64.2" will be created in ~/.cache/virt-builder/ (.2 == revision 2) I tried various experiments, and it seems as if the Xen modules are included in the initramfs even if you upgrade the kernel when running virt-builder itself. So all looks good over here, but I didn't actually try booting the disk image on Xen itself. While I have your attention ... It'd be great if libguestfs could use Xen as a backend (in addition to current qemu, KVM and UML backends). Most likely that would involve one of two approaches: (1) Modify the libvirt backend (src/launch-libvirt.c) so it works properly when the hypervisor is Xen. If you buy into libvirt, this one is probably going to be less code. (2) Add a new backend (src/launch-xl.c ?) which uses native Xen APIs to create the appliance. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Thu, Jun 12, 2014 at 4:13 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, Jun 12, 2014 at 03:12:46PM +0100, Richard W.M. Jones wrote:
Not quite as easy as I imagined. However by using the attached patch I have managed to add the two Xen drivers to the initramfs. (I haven't been able to test that it boots in Xen however)
I'm just building an updated Fedora 20 image which I'll upload later today.
The F20 image has been uploaded. You should notice when you install it that it will download the full disk image again, and also that a file "fedora-20.x86_64.2" will be created in ~/.cache/virt-builder/ (.2 == revision 2)
I tried various experiments, and it seems as if the Xen modules are included in the initramfs even if you upgrade the kernel when running virt-builder itself. So all looks good over here, but I didn't actually try booting the disk image on Xen itself.
Right, well after managing to work around our corporate transparent caching proxy to actually get the new version, I've tested it and it works great. Thanks.
While I have your attention ...
It'd be great if libguestfs could use Xen as a backend (in addition to current qemu, KVM and UML backends). Most likely that would involve one of two approaches:
(1) Modify the libvirt backend (src/launch-libvirt.c) so it works properly when the hypervisor is Xen. If you buy into libvirt, this one is probably going to be less code.
(2) Add a new backend (src/launch-xl.c ?) which uses native Xen APIs to create the appliance.
Considerations: * libvirt has had basic libxl support for some time. I doubt you're doing advanced things like migration or PCI pass-through, so I think using a recent libvirt could be made to work without too much difficulty. * Depending on exactly what features you use, libvirt doesn't actually abstract away the hypervisor details: you may need to have specific options for Xen anyway. For example, you'll need to ask for Xen devices rather than virtio devices. * Still, it's probably a lot less code / testing to just use libvirt. * But, it introduces another dependency that you need to have installed in order to use virt-builder on xen. * On the other hand, I suspect most distros will have a libguestfs->libvirt dependency anyway. So it seems like going with #1 is probably the best, overall. -George
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
On Thu, Jun 12, 2014 at 07:04:06PM +0100, George Dunlap wrote:
Right, well after managing to work around our corporate transparent caching proxy to actually get the new version, I've tested it and it works great. Thanks.
Thanks for testing. I'll make the same change to CentOS, but that will take a bit longer to build & upload.
On Thu, Jun 12, 2014 at 4:13 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
While I have your attention ...
It'd be great if libguestfs could use Xen as a backend (in addition to current qemu, KVM and UML backends). Most likely that would involve one of two approaches:
(1) Modify the libvirt backend (src/launch-libvirt.c) so it works properly when the hypervisor is Xen. If you buy into libvirt, this one is probably going to be less code.
(2) Add a new backend (src/launch-xl.c ?) which uses native Xen APIs to create the appliance.
Considerations:
* libvirt has had basic libxl support for some time. I doubt you're doing advanced things like migration or PCI pass-through, so I think using a recent libvirt could be made to work without too much difficulty.
* Depending on exactly what features you use, libvirt doesn't actually abstract away the hypervisor details: you may need to have specific options for Xen anyway. For example, you'll need to ask for Xen devices rather than virtio devices.
We don't do anything complicated at all. However definitely different from KVM assumptions made by the current libvirt backend. The current libvirt backend is known not to work with Xen (in fact, it includes code to tell you that if you try it). Here is the libvirt backend code: https://github.com/libguestfs/libguestfs/blob/master/src/launch-libvirt.c XML generation starts on line ~826.
* Still, it's probably a lot less code / testing to just use libvirt.
Yes, I definitely think #1 is going to be less code.
* But, it introduces another dependency that you need to have installed in order to use virt-builder on xen.
* On the other hand, I suspect most distros will have a libguestfs->libvirt dependency anyway.
Upstream, I have steered away from using the libvirt backend. Running qemu directly is the default. The reason is that the libvirt backend adds complexity without a great deal of benefit for most users. [That's less true if we're talking about supporting Xen because there it gives us benefit by moving much of the Xen support code into libvirt.] In Fedora & RHEL we are using the libvirt backend by default, partly because me & Dan Berrange are around to support it, and partly because it allows us to use SELinux + cgroups to contain the qemu process (somewhat analogous to driver-domains in Xen). AFAIK no other distro uses the libvirt backend by default, but they all ship it and you can switch to it by setting an environment variable (whether it actually works or not is another question).
So it seems like going with #1 is probably the best, overall.
I'm not going to implement the Xen backend myself, but patches welcome as they say. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Thu, Jun 12, 2014 at 7:27 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, Jun 12, 2014 at 07:04:06PM +0100, George Dunlap wrote:
Right, well after managing to work around our corporate transparent caching proxy to actually get the new version, I've tested it and it works great. Thanks.
Thanks for testing. I'll make the same change to CentOS, but that will take a bit longer to build & upload.
The CentOS 6 and Scientific Linux 6 templates work fine already for some reason. In fact, I've now tested all the templates, and they *all* have the proper drivers, with the exception of fedora-19. (And of course rhel-7rc, which I guess will be going away soon now that RHEL 7 has been released.) fedora-18, all the ubuntus, all the debians, centos-6 and scientificlinux-6 all boot out-of-the-box already.
* On the other hand, I suspect most distros will have a libguestfs->libvirt dependency anyway.
Upstream, I have steered away from using the libvirt backend. Running qemu directly is the default. The reason is that the libvirt backend adds complexity without a great deal of benefit for most users. [That's less true if we're talking about supporting Xen because there it gives us benefit by moving much of the Xen support code into libvirt.]
In Fedora & RHEL we are using the libvirt backend by default, partly because me & Dan Berrange are around to support it, and partly because it allows us to use SELinux + cgroups to contain the qemu process (somewhat analogous to driver-domains in Xen).
AFAIK no other distro uses the libvirt backend by default, but they all ship it and you can switch to it by setting an environment variable (whether it actually works or not is another question).
So it seems like going with #1 is probably the best, overall.
I'm not going to implement the Xen backend myself, but patches welcome as they say.
Of course. :-) Well I'll put it on a list of things to take a look at. Thanks for your help, -George
On Thu, Jun 12, 2014 at 08:25:30PM +0100, George Dunlap wrote:
On Thu, Jun 12, 2014 at 7:27 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Thu, Jun 12, 2014 at 07:04:06PM +0100, George Dunlap wrote:
Right, well after managing to work around our corporate transparent caching proxy to actually get the new version, I've tested it and it works great. Thanks.
Thanks for testing. I'll make the same change to CentOS, but that will take a bit longer to build & upload.
The CentOS 6 and Scientific Linux 6 templates work fine already for some reason.
In fact, I've now tested all the templates, and they *all* have the proper drivers, with the exception of fedora-19. (And of course rhel-7rc, which I guess will be going away soon now that RHEL 7 has been released.)
Great! Lucky I didn't start on the CentOS templates. I'll do the F19 one shortly. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Thu, Jun 12, 2014 at 3:12 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
Not quite as easy as I imagined. However by using the attached patch I have managed to add the two Xen drivers to the initramfs. (I haven't been able to test that it boots in Xen however)
I'm just building an updated Fedora 20 image which I'll upload later today.
Are there other guests that need to be updated in the same way?
Is there an easier way to tell than just trying them all? :-) Assuming there's not, I'll do that as a background task, and let you know what I find. (That may be next week sometime.) Thanks, -George
On Thu, Jun 12, 2014 at 04:37:56PM +0100, George Dunlap wrote:
On Thu, Jun 12, 2014 at 3:12 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
Not quite as easy as I imagined. However by using the attached patch I have managed to add the two Xen drivers to the initramfs. (I haven't been able to test that it boots in Xen however)
I'm just building an updated Fedora 20 image which I'll upload later today.
Are there other guests that need to be updated in the same way?
Is there an easier way to tell than just trying them all? :-)
Assuming there's not, I'll do that as a background task, and let you know what I find. (That may be next week sometime.)
Actually best thing to do now is to check the Fedora 20 image works on Xen. If it doesn't, then we'll have to go back to the drawing board. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
participants (4)
-
Dario Faggioli -
George Dunlap -
Olaf Hering -
Richard W.M. Jones