4:33 p.m.
Well you can all read the thread here for the technical details:
http://lists.gnu.org/archive/html/qemu-devel/2010-08/threads.html#00133
but the bottom line for anyone who wasn't in that discussion or on IRC
last week is that we have to change the way that the appliance is
built in libguestfs so that we don't depend (as much) on the qemu
-initrd option. The bad thing is this is loads of make-work at a
particularly inconvenient time. The good thing is that it should make
libguestfs 'boot' in under 5 seconds, even if your hardware is old and
slow. It will also reduce the memory and I/O requirements for using
libguestfs.
Because this issue is adversely affecting people using kernel 2.6.35
and/or the latest qemu right now (and will affect many people when
Fedora 14 is released which has both), this is my top priority to fix
in the coming week.
This email documents how I will change appliance building. The
patches will follow some time later in the week when I've written and
tested them.
The new boot method will be this sequence:
(a) (Same as now) A suitable kernel to use is located on the host, or
from $libdir/guestfs in the non-supermin case.
(b) A tiny initrd is built (on the fly for supermin). This will
contain just these files:
any CD-ROM, IDE, virtio kmods required to read the virtual CD device
ext2.ko if required
/sbin/modprobe
a tiny init script, written in C
(c) An ext2 image is created for the root filesystem. I will say a
lot more about how this is created below.
(d) qemu is invoked with something along these lines:
qemu -kernel kernel -initrd initrd -drive file=isofile,media=cdrom
where:
kernel = kernel found in step (a)
initrd = tiny initrd created in step (b)
isofile = ext2 filesystem image created in step (c)
(e) The boot process proceeds by starting the kernel, reading the
/init script from the tiny initrd which mounts the filesystem from the
CD-ROM device and pivot_root(2)s into it, running another /init script
from that filesystem. At this point, boot continues as it does in the
current libguestfs.
Since this involves attaching an extra device to the appliance, we
also need to change the daemon to ignore this extra device, adding
somewhat to the complexity of several operations.
We believe this should improve the speed of boot greatly. Obviously
there is the saving because we are no longer using the (now broken)
qemu -initrd support with a large initrd, but that's purely because of
a qemu regression. We also save because we don't need to unpack the
initrd inside the appliance, and because the device is loaded on
demand. Also there are some unrelated changes that I intend to make
which will improve boot speeds. We have every reason to believe that
5 seconds will be achievable, even on relatively old hardware.
Step (c) above involves creating an ext2 filesystem for root. I chose
ext2 because it is considerably less complex than ISO9660, and is the
native Linux filesystem so it supports all Linux features (long
filenames, extended attributes and so on).
Creating an ext2 filesystem is more complex than the initrd that we
currently create using some hand-coded cpio-like code. Of course we
can't use libguestfs to help us! We plan to write some C code to
assemble the ext2 filesystem from scratch (or starting from a mke2fs
blank filesystem).
To further improve boot times, we intend to cache this so it will only
be created the first time it is used, and only when it needs to be
updated. Caching this means that most times that you use libguestfs
or other tools, no appliance creation will be required at all, and
only the bits of the appliance that you use will be loaded.
If you have any further questions about this, please follow up on this
mailing list.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
8:29 p.m.
Can the applicance support virtio-net and start a ssh server?
And if it support xen-hvm mode, it is much more useful.
Regards
Kirby Zhou
-----Original Message-----
From: libguestfs-bounces(a)redhat.com [mailto:libguestfs-bounces@redhat.com]
On Behalf Of Richard W.M. Jones
Sent: Saturday, August 14, 2010 5:34 AM
To: libguestfs(a)redhat.com
Subject: [Libguestfs] Plans for changing libguestfs appliance building
Well you can all read the thread here for the technical details:
http://lists.gnu.org/archive/html/qemu-devel/2010-08/threads.html#00133
but the bottom line for anyone who wasn't in that discussion or on IRC
last week is that we have to change the way that the appliance is
built in libguestfs so that we don't depend (as much) on the qemu
-initrd option. The bad thing is this is loads of make-work at a
particularly inconvenient time. The good thing is that it should make
libguestfs 'boot' in under 5 seconds, even if your hardware is old and
slow. It will also reduce the memory and I/O requirements for using
libguestfs.
Because this issue is adversely affecting people using kernel 2.6.35
and/or the latest qemu right now (and will affect many people when
Fedora 14 is released which has both), this is my top priority to fix
in the coming week.
This email documents how I will change appliance building. The
patches will follow some time later in the week when I've written and
tested them.
The new boot method will be this sequence:
(a) (Same as now) A suitable kernel to use is located on the host, or
from $libdir/guestfs in the non-supermin case.
(b) A tiny initrd is built (on the fly for supermin). This will
contain just these files:
any CD-ROM, IDE, virtio kmods required to read the virtual CD device
ext2.ko if required
/sbin/modprobe
a tiny init script, written in C
(c) An ext2 image is created for the root filesystem. I will say a
lot more about how this is created below.
(d) qemu is invoked with something along these lines:
qemu -kernel kernel -initrd initrd -drive file=isofile,media=cdrom
where:
kernel = kernel found in step (a)
initrd = tiny initrd created in step (b)
isofile = ext2 filesystem image created in step (c)
(e) The boot process proceeds by starting the kernel, reading the
/init script from the tiny initrd which mounts the filesystem from the
CD-ROM device and pivot_root(2)s into it, running another /init script
from that filesystem. At this point, boot continues as it does in the
current libguestfs.
Since this involves attaching an extra device to the appliance, we
also need to change the daemon to ignore this extra device, adding
somewhat to the complexity of several operations.
We believe this should improve the speed of boot greatly. Obviously
there is the saving because we are no longer using the (now broken)
qemu -initrd support with a large initrd, but that's purely because of
a qemu regression. We also save because we don't need to unpack the
initrd inside the appliance, and because the device is loaded on
demand. Also there are some unrelated changes that I intend to make
which will improve boot speeds. We have every reason to believe that
5 seconds will be achievable, even on relatively old hardware.
Step (c) above involves creating an ext2 filesystem for root. I chose
ext2 because it is considerably less complex than ISO9660, and is the
native Linux filesystem so it supports all Linux features (long
filenames, extended attributes and so on).
Creating an ext2 filesystem is more complex than the initrd that we
currently create using some hand-coded cpio-like code. Of course we
can't use libguestfs to help us! We plan to write some C code to
assemble the ext2 filesystem from scratch (or starting from a mke2fs
blank filesystem).
To further improve boot times, we intend to cache this so it will only
be created the first time it is used, and only when it needs to be
updated. Caching this means that most times that you use libguestfs
or other tools, no appliance creation will be required at all, and
only the bits of the appliance that you use will be loaded.
If you have any further questions about this, please follow up on this
mailing list.
Rich.
--
Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
_______________________________________________
Libguestfs mailing list
Libguestfs(a)redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
4:43 p.m.
On Sun, Aug 15, 2010 at 09:29:29AM +0800, Kirby Zhou wrote:
Can the applicance support virtio-net and start a ssh server?
This is actually a really important point which I'd overlooked. If we
disable network support completely then the appliance will have no
network, which will negatively impact things like virt-v2v and anyone
trying to run any network-aware commands.
On the other hand, it's desirable to disable network support for most
operations because of the speed up that is possible. Also it's
possible that qemu will be configured with no user-networking (slirp)
at all.
We'll need to have some sort of flag to enable/disable/detect this,
and treat network support as a separable feature. At the moment I
don't know exactly how this will work.
And if it support xen-hvm mode, it is much more useful.
I don't understand this bit.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
9:17 p.m.
On 08/14/2010 05:33 AM, Richard W.M. Jones wrote:
Well you can all read the thread here for the technical details:
http://lists.gnu.org/archive/html/qemu-devel/2010-08/threads.html#00133
but the bottom line for anyone who wasn't in that discussion or on IRC
last week is that we have to change the way that the appliance is
built in libguestfs so that we don't depend (as much) on the qemu
-initrd option. The bad thing is this is loads of make-work at a
particularly inconvenient time. The good thing is that it should make
libguestfs 'boot' in under 5 seconds, even if your hardware is old and
slow. It will also reduce the memory and I/O requirements for using
libguestfs.
Because this issue is adversely affecting people using kernel 2.6.35
and/or the latest qemu right now (and will affect many people when
Fedora 14 is released which has both), this is my top priority to fix
in the coming week.
This email documents how I will change appliance building. The
patches will follow some time later in the week when I've written and
tested them.
The new boot method will be this sequence:
(a) (Same as now) A suitable kernel to use is located on the host, or
from $libdir/guestfs in the non-supermin case.
(b) A tiny initrd is built (on the fly for supermin). This will
contain just these files:
any CD-ROM, IDE, virtio kmods required to read the virtual CD device
ext2.ko if required
/sbin/modprobe
a tiny init script, written in C
(c) An ext2 image is created for the root filesystem. I will say a
lot more about how this is created below.
(d) qemu is invoked with something along these lines:
qemu -kernel kernel -initrd initrd -drive file=isofile,media=cdrom
where:
kernel = kernel found in step (a)
initrd = tiny initrd created in step (b)
isofile = ext2 filesystem image created in step (c)
(e) The boot process proceeds by starting the kernel, reading the
/init script from the tiny initrd which mounts the filesystem from the
CD-ROM device and pivot_root(2)s into it, running another /init script
from that filesystem. At this point, boot continues as it does in the
current libguestfs.
The virtual cdrom device in qemu will be readonly (is that right?).
Then how are you going to tune the boot process as I know it would
require writing to the filesystem.
Since this involves attaching an extra device to the appliance, we
also need to change the daemon to ignore this extra device, adding
somewhat to the complexity of several operations.
We believe this should improve the speed of boot greatly. Obviously
there is the saving because we are no longer using the (now broken)
qemu -initrd support with a large initrd, but that's purely because of
a qemu regression. We also save because we don't need to unpack the
initrd inside the appliance, and because the device is loaded on
demand. Also there are some unrelated changes that I intend to make
which will improve boot speeds. We have every reason to believe that
5 seconds will be achievable, even on relatively old hardware.
Step (c) above involves creating an ext2 filesystem for root. I chose
ext2 because it is considerably less complex than ISO9660, and is the
native Linux filesystem so it supports all Linux features (long
filenames, extended attributes and so on).
Creating an ext2 filesystem is more complex than the initrd that we
currently create using some hand-coded cpio-like code. Of course we
can't use libguestfs to help us! We plan to write some C code to
assemble the ext2 filesystem from scratch (or starting from a mke2fs
blank filesystem).
To further improve boot times, we intend to cache this so it will only
be created the first time it is used, and only when it needs to be
updated. Caching this means that most times that you use libguestfs
or other tools, no appliance creation will be required at all, and
only the bits of the appliance that you use will be loaded.
If you have any further questions about this, please follow up on this
mailing list.
Rich.
--
Regards.
Jinxin
5:40 a.m.
On Mon, Aug 16, 2010 at 10:17:30AM +0800, Jin Xin Zheng wrote:
The virtual cdrom device in qemu will be readonly (is that right?).
Then how are you going to tune the boot process as I know it would
require writing to the filesystem.
I'm not aware of us needing to write to the root filesystem during
boot (well, there is one place where we do 'mkdir -p /sysroot' but
that can be fixed).
However there is an issue that we modify configuration files in the
daemon, in particular /etc/lvm/lvm.conf, which would require either
making this directory a tmpfs or using some other method to make root
writable.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
6:21 a.m.
On 08/16/2010 06:40 PM, Richard W.M. Jones wrote:
On Mon, Aug 16, 2010 at 10:17:30AM +0800, Jin Xin Zheng wrote:
> The virtual cdrom device in qemu will be readonly (is that right?).
>
> Then how are you going to tune the boot process as I know it would
> require writing to the filesystem.
>
I'm not aware of us needing to write to the root filesystem during
boot (well, there is one place where we do 'mkdir -p /sysroot' but
that can be fixed).
However there is an issue that we modify configuration files in the
daemon, in particular /etc/lvm/lvm.conf, which would require either
making this directory a tmpfs or using some other method to make root
writable.
Rich.
Are you sure all the binary utils inside the appliance will not write
logging files when running?
In fact when I saw this the first thing came up to my head was that our
modified source for doing gcov -- we could collect the .gcda files in
the build directory after libguestfs was invoked. But the guestfsd which
runs inside the appliance produces the .gcda files in the temp ramfs.
We have made a pretty lot of effort in order to tar-out these files
before the subprocess was killed.
If the filesystem is readonly then I'll have to do more to allow guestfsd
to write debugging coverage info into a writable place. However this is
not a problem for libguestfs, it's just a problem for me...
--
Regards.
Jinxin
7:46 a.m.
On Mon, Aug 16, 2010 at 07:21:12PM +0800, Jin Xin Zheng wrote:
On 08/16/2010 06:40 PM, Richard W.M. Jones wrote:
>On Mon, Aug 16, 2010 at 10:17:30AM +0800, Jin Xin Zheng wrote:
>>The virtual cdrom device in qemu will be readonly (is that right?).
>>
>>Then how are you going to tune the boot process as I know it would
>>require writing to the filesystem.
>I'm not aware of us needing to write to the root filesystem during
>boot (well, there is one place where we do 'mkdir -p /sysroot' but
>that can be fixed).
>
>However there is an issue that we modify configuration files in the
>daemon, in particular /etc/lvm/lvm.conf, which would require either
>making this directory a tmpfs or using some other method to make root
>writable.
>
>Rich.
>
Are you sure all the binary utils inside the appliance will not write
logging files when running?
I guess we can mount /tmp, /var and maybe a few other things as tmpfs?
In fact when I saw this the first thing came up to my head was that
our
modified source for doing gcov -- we could collect the .gcda files in
the build directory after libguestfs was invoked. But the guestfsd which
runs inside the appliance produces the .gcda files in the temp ramfs.
We have made a pretty lot of effort in order to tar-out these files
before the subprocess was killed.
If the filesystem is readonly then I'll have to do more to allow guestfsd
to write debugging coverage info into a writable place. However this is
not a problem for libguestfs, it's just a problem for me...
The alternative is to use a regular drive (not a CD), and attach it
using:
-drive file=isofile,snapshot=on
Since this will be one of the drives, it has two disadvantages: we
need to work a little bit harder to hide this drive from users, and it
reduces the number of drives available to users from 26 to 25
(assuming drives can only be named /dev/[hsv]d[a-z], which is true of a
lot of daemon code at the moment).
Live CDs use another method: They create a device mapper writable
snapshot over the filesystem, allowing you to write to the normally
read-only CD. I don't think this will work for us since we have to be
extra careful not to use device mapper names, LVs etc which might
conflict with any on the user drives. Although I've not thought this
through very much, so perhaps there is some way.
I think it's fair to say also that we're not switching away from
initrd because we want to. It's because we are forced to ...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
9:05 a.m.
On Mon, Aug 16, 2010 at 01:46:13PM +0100, Richard W.M. Jones wrote:
On Mon, Aug 16, 2010 at 07:21:12PM +0800, Jin Xin Zheng wrote:
> On 08/16/2010 06:40 PM, Richard W.M. Jones wrote:
> >On Mon, Aug 16, 2010 at 10:17:30AM +0800, Jin Xin Zheng wrote:
> >>The virtual cdrom device in qemu will be readonly (is that right?).
> >>
> >>Then how are you going to tune the boot process as I know it would
> >>require writing to the filesystem.
> >I'm not aware of us needing to write to the root filesystem during
> >boot (well, there is one place where we do 'mkdir -p /sysroot' but
> >that can be fixed).
> >
> >However there is an issue that we modify configuration files in the
> >daemon, in particular /etc/lvm/lvm.conf, which would require either
> >making this directory a tmpfs or using some other method to make root
> >writable.
> >
> >Rich.
> >
> Are you sure all the binary utils inside the appliance will not write
> logging files when running?
I guess we can mount /tmp, /var and maybe a few other things as tmpfs?
> In fact when I saw this the first thing came up to my head was that our
> modified source for doing gcov -- we could collect the .gcda files in
> the build directory after libguestfs was invoked. But the guestfsd which
> runs inside the appliance produces the .gcda files in the temp ramfs.
> We have made a pretty lot of effort in order to tar-out these files
> before the subprocess was killed.
>
> If the filesystem is readonly then I'll have to do more to allow guestfsd
> to write debugging coverage info into a writable place. However this is
> not a problem for libguestfs, it's just a problem for me...
The alternative is to use a regular drive (not a CD), and attach it
using:
-drive file=isofile,snapshot=on
Since this will be one of the drives, it has two disadvantages: we
need to work a little bit harder to hide this drive from users, and it
reduces the number of drives available to users from 26 to 25
(assuming drives can only be named /dev/[hsv]d[a-z], which is true of a
lot of daemon code at the moment).
I don't know the internals of libguestfs that well, but is filtering
out a harddisk really more trouble than filtering out a CDROM ?
Any IDE/SATA CDROM device will appear as /dev/[hsv]d[a-z] too, even
if it has a /dev/cdrom link. So AFAICT you need todo the same kind
of filtering for either disks or cdroms, and you loose 1 drive letter
in both cases :-(
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
9:39 a.m.
On Mon, Aug 16, 2010 at 03:05:22PM +0100, Daniel P. Berrange wrote:
I don't know the internals of libguestfs that well, but is
filtering
out a harddisk really more trouble than filtering out a CDROM ?
Any IDE/SATA CDROM device will appear as /dev/[hsv]d[a-z] too, even
if it has a /dev/cdrom link. So AFAICT you need todo the same kind
of filtering for either disks or cdroms, and you loose 1 drive letter
in both cases :-(
Yes, thinking about this more, I think this is true.
Note we have to add it as the last drive anyway (since everyone
assumes the first drive added is /dev/sda). Which means it doesn't
have a fixed name, so we have to pass the name in as a kernel command
line parameter, so the small initrd knows what to mount.
Matt has also mentioned using udev to rename devices, and also that
this might be a good time to add some swap.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
8:06 a.m.
On Mon, Aug 16, 2010 at 07:21:12PM +0800, Jin Xin Zheng wrote:
In fact when I saw this the first thing came up to my head was that
our
modified source for doing gcov -- we could collect the .gcda files in
the build directory after libguestfs was invoked. But the guestfsd which
runs inside the appliance produces the .gcda files in the temp ramfs.
I guess it writes it into / (ie. guestfsd's current directory which is
pretty much always root)?
We have made a pretty lot of effort in order to tar-out these files
before the subprocess was killed.
We have/had the same problem capturing core files too ...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
9:13 p.m.
On 08/16/2010 09:06 PM, Richard W.M. Jones wrote:
On Mon, Aug 16, 2010 at 07:21:12PM +0800, Jin Xin Zheng wrote:
> In fact when I saw this the first thing came up to my head was that our
> modified source for doing gcov -- we could collect the .gcda files in
> the build directory after libguestfs was invoked. But the guestfsd which
> runs inside the appliance produces the .gcda files in the temp ramfs.
>
I guess it writes it into / (ie. guestfsd's current directory which is
pretty much always root)?
Yes you are right. gcov does not care about the $pwd, it just outputs
.gcda files under the directory where I built the source, e.g.
/home/build/rpmbuild/BUILD/libguestfs-1.2.7/daemon
In order to download these files correctly, we have to use the debug
command to access the root filesystem, although building the source in a
directory under /sysroot may make things easier.
> We have made a pretty lot of effort in order to tar-out these
files
> before the subprocess was killed.
>
We have/had the same problem capturing core files too ...
Rich.
--
Regards.
Jinxin
2:18 a.m.
On Tue, Aug 17, 2010 at 10:13:59AM +0800, Jinxin Zheng wrote:
On 08/16/2010 09:06 PM, Richard W.M. Jones wrote:
>On Mon, Aug 16, 2010 at 07:21:12PM +0800, Jin Xin Zheng wrote:
>>In fact when I saw this the first thing came up to my head was that our
>>modified source for doing gcov -- we could collect the .gcda files in
>>the build directory after libguestfs was invoked. But the guestfsd which
>>runs inside the appliance produces the .gcda files in the temp ramfs.
>I guess it writes it into / (ie. guestfsd's current directory which is
>pretty much always root)?
>
Yes you are right. gcov does not care about the $pwd, it just outputs
.gcda files under the directory where I built the source, e.g.
/home/build/rpmbuild/BUILD/libguestfs-1.2.7/daemon
In order to download these files correctly, we have to use the debug
command to access the root filesystem, although building the source in a
directory under /sysroot may make things easier.
It looks like we'll be using a writable root device anyway, so
problem solved ...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/
5212
days inactive
5216
days old
11 comments
5 participants
participants (5)
-
Daniel P. Berrange -
Jin Xin Zheng -
Jinxin Zheng
-
Kirby Zhou -
Richard W.M. Jones