Plans for changing libguestfs appliance building
Well you can all read the thread here for the technical details: http://lists.gnu.org/archive/html/qemu-devel/2010-08/threads.html#00133 but the bottom line for anyone who wasn't in that discussion or on IRC last week is that we have to change the way that the appliance is built in libguestfs so that we don't depend (as much) on the qemu -initrd option. The bad thing is this is loads of make-work at a particularly inconvenient time. The good thing is that it should make libguestfs 'boot' in under 5 seconds, even if your hardware is old and slow. It will also reduce the memory and I/O requirements for using libguestfs. Because this issue is adversely affecting people using kernel 2.6.35 and/or the latest qemu right now (and will affect many people when Fedora 14 is released which has both), this is my top priority to fix in the coming week. This email documents how I will change appliance building. The patches will follow some time later in the week when I've written and tested them. The new boot method will be this sequence: (a) (Same as now) A suitable kernel to use is located on the host, or from $libdir/guestfs in the non-supermin case. (b) A tiny initrd is built (on the fly for supermin). This will contain just these files: any CD-ROM, IDE, virtio kmods required to read the virtual CD device ext2.ko if required /sbin/modprobe a tiny init script, written in C (c) An ext2 image is created for the root filesystem. I will say a lot more about how this is created below. (d) qemu is invoked with something along these lines: qemu -kernel kernel -initrd initrd -drive file=isofile,media=cdrom where: kernel = kernel found in step (a) initrd = tiny initrd created in step (b) isofile = ext2 filesystem image created in step (c) (e) The boot process proceeds by starting the kernel, reading the /init script from the tiny initrd which mounts the filesystem from the CD-ROM device and pivot_root(2)s into it, running another /init script from that filesystem. At this point, boot continues as it does in the current libguestfs. Since this involves attaching an extra device to the appliance, we also need to change the daemon to ignore this extra device, adding somewhat to the complexity of several operations. We believe this should improve the speed of boot greatly. Obviously there is the saving because we are no longer using the (now broken) qemu -initrd support with a large initrd, but that's purely because of a qemu regression. We also save because we don't need to unpack the initrd inside the appliance, and because the device is loaded on demand. Also there are some unrelated changes that I intend to make which will improve boot speeds. We have every reason to believe that 5 seconds will be achievable, even on relatively old hardware. Step (c) above involves creating an ext2 filesystem for root. I chose ext2 because it is considerably less complex than ISO9660, and is the native Linux filesystem so it supports all Linux features (long filenames, extended attributes and so on). Creating an ext2 filesystem is more complex than the initrd that we currently create using some hand-coded cpio-like code. Of course we can't use libguestfs to help us! We plan to write some C code to assemble the ext2 filesystem from scratch (or starting from a mke2fs blank filesystem). To further improve boot times, we intend to cache this so it will only be created the first time it is used, and only when it needs to be updated. Caching this means that most times that you use libguestfs or other tools, no appliance creation will be required at all, and only the bits of the appliance that you use will be loaded. If you have any further questions about this, please follow up on this mailing list. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
Can the applicance support virtio-net and start a ssh server? And if it support xen-hvm mode, it is much more useful. Regards Kirby Zhou -----Original Message----- From: libguestfs-bounces@redhat.com [mailto:libguestfs-bounces@redhat.com] On Behalf Of Richard W.M. Jones Sent: Saturday, August 14, 2010 5:34 AM To: libguestfs@redhat.com Subject: [Libguestfs] Plans for changing libguestfs appliance building Well you can all read the thread here for the technical details: http://lists.gnu.org/archive/html/qemu-devel/2010-08/threads.html#00133 but the bottom line for anyone who wasn't in that discussion or on IRC last week is that we have to change the way that the appliance is built in libguestfs so that we don't depend (as much) on the qemu -initrd option. The bad thing is this is loads of make-work at a particularly inconvenient time. The good thing is that it should make libguestfs 'boot' in under 5 seconds, even if your hardware is old and slow. It will also reduce the memory and I/O requirements for using libguestfs. Because this issue is adversely affecting people using kernel 2.6.35 and/or the latest qemu right now (and will affect many people when Fedora 14 is released which has both), this is my top priority to fix in the coming week. This email documents how I will change appliance building. The patches will follow some time later in the week when I've written and tested them. The new boot method will be this sequence: (a) (Same as now) A suitable kernel to use is located on the host, or from $libdir/guestfs in the non-supermin case. (b) A tiny initrd is built (on the fly for supermin). This will contain just these files: any CD-ROM, IDE, virtio kmods required to read the virtual CD device ext2.ko if required /sbin/modprobe a tiny init script, written in C (c) An ext2 image is created for the root filesystem. I will say a lot more about how this is created below. (d) qemu is invoked with something along these lines: qemu -kernel kernel -initrd initrd -drive file=isofile,media=cdrom where: kernel = kernel found in step (a) initrd = tiny initrd created in step (b) isofile = ext2 filesystem image created in step (c) (e) The boot process proceeds by starting the kernel, reading the /init script from the tiny initrd which mounts the filesystem from the CD-ROM device and pivot_root(2)s into it, running another /init script from that filesystem. At this point, boot continues as it does in the current libguestfs. Since this involves attaching an extra device to the appliance, we also need to change the daemon to ignore this extra device, adding somewhat to the complexity of several operations. We believe this should improve the speed of boot greatly. Obviously there is the saving because we are no longer using the (now broken) qemu -initrd support with a large initrd, but that's purely because of a qemu regression. We also save because we don't need to unpack the initrd inside the appliance, and because the device is loaded on demand. Also there are some unrelated changes that I intend to make which will improve boot speeds. We have every reason to believe that 5 seconds will be achievable, even on relatively old hardware. Step (c) above involves creating an ext2 filesystem for root. I chose ext2 because it is considerably less complex than ISO9660, and is the native Linux filesystem so it supports all Linux features (long filenames, extended attributes and so on). Creating an ext2 filesystem is more complex than the initrd that we currently create using some hand-coded cpio-like code. Of course we can't use libguestfs to help us! We plan to write some C code to assemble the ext2 filesystem from scratch (or starting from a mke2fs blank filesystem). To further improve boot times, we intend to cache this so it will only be created the first time it is used, and only when it needs to be updated. Caching this means that most times that you use libguestfs or other tools, no appliance creation will be required at all, and only the bits of the appliance that you use will be loaded. If you have any further questions about this, please follow up on this mailing list. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
On Sun, Aug 15, 2010 at 09:29:29AM +0800, Kirby Zhou wrote:
Can the applicance support virtio-net and start a ssh server?
This is actually a really important point which I'd overlooked. If we disable network support completely then the appliance will have no network, which will negatively impact things like virt-v2v and anyone trying to run any network-aware commands. On the other hand, it's desirable to disable network support for most operations because of the speed up that is possible. Also it's possible that qemu will be configured with no user-networking (slirp) at all. We'll need to have some sort of flag to enable/disable/detect this, and treat network support as a separable feature. At the moment I don't know exactly how this will work.
And if it support xen-hvm mode, it is much more useful.
I don't understand this bit. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On 08/14/2010 05:33 AM, Richard W.M. Jones wrote:
Well you can all read the thread here for the technical details: http://lists.gnu.org/archive/html/qemu-devel/2010-08/threads.html#00133 but the bottom line for anyone who wasn't in that discussion or on IRC last week is that we have to change the way that the appliance is built in libguestfs so that we don't depend (as much) on the qemu -initrd option. The bad thing is this is loads of make-work at a particularly inconvenient time. The good thing is that it should make libguestfs 'boot' in under 5 seconds, even if your hardware is old and slow. It will also reduce the memory and I/O requirements for using libguestfs.
Because this issue is adversely affecting people using kernel 2.6.35 and/or the latest qemu right now (and will affect many people when Fedora 14 is released which has both), this is my top priority to fix in the coming week.
This email documents how I will change appliance building. The patches will follow some time later in the week when I've written and tested them.
The new boot method will be this sequence:
(a) (Same as now) A suitable kernel to use is located on the host, or from $libdir/guestfs in the non-supermin case.
(b) A tiny initrd is built (on the fly for supermin). This will contain just these files:
any CD-ROM, IDE, virtio kmods required to read the virtual CD device ext2.ko if required /sbin/modprobe a tiny init script, written in C
(c) An ext2 image is created for the root filesystem. I will say a lot more about how this is created below.
(d) qemu is invoked with something along these lines:
qemu -kernel kernel -initrd initrd -drive file=isofile,media=cdrom
where: kernel = kernel found in step (a) initrd = tiny initrd created in step (b) isofile = ext2 filesystem image created in step (c)
(e) The boot process proceeds by starting the kernel, reading the /init script from the tiny initrd which mounts the filesystem from the CD-ROM device and pivot_root(2)s into it, running another /init script from that filesystem. At this point, boot continues as it does in the current libguestfs.
The virtual cdrom device in qemu will be readonly (is that right?). Then how are you going to tune the boot process as I know it would require writing to the filesystem.
Since this involves attaching an extra device to the appliance, we also need to change the daemon to ignore this extra device, adding somewhat to the complexity of several operations.
We believe this should improve the speed of boot greatly. Obviously there is the saving because we are no longer using the (now broken) qemu -initrd support with a large initrd, but that's purely because of a qemu regression. We also save because we don't need to unpack the initrd inside the appliance, and because the device is loaded on demand. Also there are some unrelated changes that I intend to make which will improve boot speeds. We have every reason to believe that 5 seconds will be achievable, even on relatively old hardware.
Step (c) above involves creating an ext2 filesystem for root. I chose ext2 because it is considerably less complex than ISO9660, and is the native Linux filesystem so it supports all Linux features (long filenames, extended attributes and so on).
Creating an ext2 filesystem is more complex than the initrd that we currently create using some hand-coded cpio-like code. Of course we can't use libguestfs to help us! We plan to write some C code to assemble the ext2 filesystem from scratch (or starting from a mke2fs blank filesystem).
To further improve boot times, we intend to cache this so it will only be created the first time it is used, and only when it needs to be updated. Caching this means that most times that you use libguestfs or other tools, no appliance creation will be required at all, and only the bits of the appliance that you use will be loaded.
If you have any further questions about this, please follow up on this mailing list.
Rich.
-- Regards. Jinxin
On Mon, Aug 16, 2010 at 10:17:30AM +0800, Jin Xin Zheng wrote:
The virtual cdrom device in qemu will be readonly (is that right?).
Then how are you going to tune the boot process as I know it would require writing to the filesystem.
I'm not aware of us needing to write to the root filesystem during boot (well, there is one place where we do 'mkdir -p /sysroot' but that can be fixed). However there is an issue that we modify configuration files in the daemon, in particular /etc/lvm/lvm.conf, which would require either making this directory a tmpfs or using some other method to make root writable. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
On 08/16/2010 06:40 PM, Richard W.M. Jones wrote:
On Mon, Aug 16, 2010 at 10:17:30AM +0800, Jin Xin Zheng wrote:
The virtual cdrom device in qemu will be readonly (is that right?).
Then how are you going to tune the boot process as I know it would require writing to the filesystem.
I'm not aware of us needing to write to the root filesystem during boot (well, there is one place where we do 'mkdir -p /sysroot' but that can be fixed).
However there is an issue that we modify configuration files in the daemon, in particular /etc/lvm/lvm.conf, which would require either making this directory a tmpfs or using some other method to make root writable.
Rich.
Are you sure all the binary utils inside the appliance will not write logging files when running? In fact when I saw this the first thing came up to my head was that our modified source for doing gcov -- we could collect the .gcda files in the build directory after libguestfs was invoked. But the guestfsd which runs inside the appliance produces the .gcda files in the temp ramfs. We have made a pretty lot of effort in order to tar-out these files before the subprocess was killed. If the filesystem is readonly then I'll have to do more to allow guestfsd to write debugging coverage info into a writable place. However this is not a problem for libguestfs, it's just a problem for me... -- Regards. Jinxin
On Mon, Aug 16, 2010 at 07:21:12PM +0800, Jin Xin Zheng wrote:
On 08/16/2010 06:40 PM, Richard W.M. Jones wrote:
The virtual cdrom device in qemu will be readonly (is that right?).
Then how are you going to tune the boot process as I know it would require writing to the filesystem. I'm not aware of us needing to write to the root filesystem during boot (well, there is one place where we do 'mkdir -p /sysroot' but
On Mon, Aug 16, 2010 at 10:17:30AM +0800, Jin Xin Zheng wrote: that can be fixed).
However there is an issue that we modify configuration files in the daemon, in particular /etc/lvm/lvm.conf, which would require either making this directory a tmpfs or using some other method to make root writable.
Rich.
Are you sure all the binary utils inside the appliance will not write logging files when running?
I guess we can mount /tmp, /var and maybe a few other things as tmpfs?
In fact when I saw this the first thing came up to my head was that our modified source for doing gcov -- we could collect the .gcda files in the build directory after libguestfs was invoked. But the guestfsd which runs inside the appliance produces the .gcda files in the temp ramfs. We have made a pretty lot of effort in order to tar-out these files before the subprocess was killed.
If the filesystem is readonly then I'll have to do more to allow guestfsd to write debugging coverage info into a writable place. However this is not a problem for libguestfs, it's just a problem for me...
The alternative is to use a regular drive (not a CD), and attach it using: -drive file=isofile,snapshot=on Since this will be one of the drives, it has two disadvantages: we need to work a little bit harder to hide this drive from users, and it reduces the number of drives available to users from 26 to 25 (assuming drives can only be named /dev/[hsv]d[a-z], which is true of a lot of daemon code at the moment). Live CDs use another method: They create a device mapper writable snapshot over the filesystem, allowing you to write to the normally read-only CD. I don't think this will work for us since we have to be extra careful not to use device mapper names, LVs etc which might conflict with any on the user drives. Although I've not thought this through very much, so perhaps there is some way. I think it's fair to say also that we're not switching away from initrd because we want to. It's because we are forced to ... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
On Mon, Aug 16, 2010 at 01:46:13PM +0100, Richard W.M. Jones wrote:
On Mon, Aug 16, 2010 at 07:21:12PM +0800, Jin Xin Zheng wrote:
On 08/16/2010 06:40 PM, Richard W.M. Jones wrote:
The virtual cdrom device in qemu will be readonly (is that right?).
Then how are you going to tune the boot process as I know it would require writing to the filesystem. I'm not aware of us needing to write to the root filesystem during boot (well, there is one place where we do 'mkdir -p /sysroot' but
On Mon, Aug 16, 2010 at 10:17:30AM +0800, Jin Xin Zheng wrote: that can be fixed).
However there is an issue that we modify configuration files in the daemon, in particular /etc/lvm/lvm.conf, which would require either making this directory a tmpfs or using some other method to make root writable.
Rich.
Are you sure all the binary utils inside the appliance will not write logging files when running?
I guess we can mount /tmp, /var and maybe a few other things as tmpfs?
In fact when I saw this the first thing came up to my head was that our modified source for doing gcov -- we could collect the .gcda files in the build directory after libguestfs was invoked. But the guestfsd which runs inside the appliance produces the .gcda files in the temp ramfs. We have made a pretty lot of effort in order to tar-out these files before the subprocess was killed.
If the filesystem is readonly then I'll have to do more to allow guestfsd to write debugging coverage info into a writable place. However this is not a problem for libguestfs, it's just a problem for me...
The alternative is to use a regular drive (not a CD), and attach it using:
-drive file=isofile,snapshot=on
Since this will be one of the drives, it has two disadvantages: we need to work a little bit harder to hide this drive from users, and it reduces the number of drives available to users from 26 to 25 (assuming drives can only be named /dev/[hsv]d[a-z], which is true of a lot of daemon code at the moment).
I don't know the internals of libguestfs that well, but is filtering out a harddisk really more trouble than filtering out a CDROM ? Any IDE/SATA CDROM device will appear as /dev/[hsv]d[a-z] too, even if it has a /dev/cdrom link. So AFAICT you need todo the same kind of filtering for either disks or cdroms, and you loose 1 drive letter in both cases :-( Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Mon, Aug 16, 2010 at 03:05:22PM +0100, Daniel P. Berrange wrote:
I don't know the internals of libguestfs that well, but is filtering out a harddisk really more trouble than filtering out a CDROM ? Any IDE/SATA CDROM device will appear as /dev/[hsv]d[a-z] too, even if it has a /dev/cdrom link. So AFAICT you need todo the same kind of filtering for either disks or cdroms, and you loose 1 drive letter in both cases :-(
Yes, thinking about this more, I think this is true. Note we have to add it as the last drive anyway (since everyone assumes the first drive added is /dev/sda). Which means it doesn't have a fixed name, so we have to pass the name in as a kernel command line parameter, so the small initrd knows what to mount. Matt has also mentioned using udev to rename devices, and also that this might be a good time to add some swap. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
On Mon, Aug 16, 2010 at 07:21:12PM +0800, Jin Xin Zheng wrote:
In fact when I saw this the first thing came up to my head was that our modified source for doing gcov -- we could collect the .gcda files in the build directory after libguestfs was invoked. But the guestfsd which runs inside the appliance produces the .gcda files in the temp ramfs.
I guess it writes it into / (ie. guestfsd's current directory which is pretty much always root)?
We have made a pretty lot of effort in order to tar-out these files before the subprocess was killed.
We have/had the same problem capturing core files too ... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On 08/16/2010 09:06 PM, Richard W.M. Jones wrote:
On Mon, Aug 16, 2010 at 07:21:12PM +0800, Jin Xin Zheng wrote:
In fact when I saw this the first thing came up to my head was that our modified source for doing gcov -- we could collect the .gcda files in the build directory after libguestfs was invoked. But the guestfsd which runs inside the appliance produces the .gcda files in the temp ramfs.
I guess it writes it into / (ie. guestfsd's current directory which is pretty much always root)?
Yes you are right. gcov does not care about the $pwd, it just outputs .gcda files under the directory where I built the source, e.g. /home/build/rpmbuild/BUILD/libguestfs-1.2.7/daemon In order to download these files correctly, we have to use the debug command to access the root filesystem, although building the source in a directory under /sysroot may make things easier.
We have made a pretty lot of effort in order to tar-out these files before the subprocess was killed.
We have/had the same problem capturing core files too ...
Rich.
-- Regards. Jinxin
On Tue, Aug 17, 2010 at 10:13:59AM +0800, Jinxin Zheng wrote:
On 08/16/2010 09:06 PM, Richard W.M. Jones wrote:
In fact when I saw this the first thing came up to my head was that our modified source for doing gcov -- we could collect the .gcda files in the build directory after libguestfs was invoked. But the guestfsd which runs inside the appliance produces the .gcda files in the temp ramfs. I guess it writes it into / (ie. guestfsd's current directory which is
On Mon, Aug 16, 2010 at 07:21:12PM +0800, Jin Xin Zheng wrote: pretty much always root)?
Yes you are right. gcov does not care about the $pwd, it just outputs .gcda files under the directory where I built the source, e.g. /home/build/rpmbuild/BUILD/libguestfs-1.2.7/daemon
In order to download these files correctly, we have to use the debug command to access the root filesystem, although building the source in a directory under /sysroot may make things easier.
It looks like we'll be using a writable root device anyway, so problem solved ... Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
participants (5)
-
Daniel P. Berrange -
Jin Xin Zheng -
Jinxin Zheng
-
Kirby Zhou -
Richard W.M. Jones