9:24 a.m.
Hi,
I remembered reading about Intel Clear Containers [1], and Hyper/Qboot [2]
that support startup times measured in the hundreds of miliseconds range.
On an AMD FX(tm)-8350 'guestfish -a /dev/null run' takes ~4s when run the 2nd/3rd
time:
real 0m4.152s
user 0m2.120s
sys 0m0.564s
Are there any plans on applying these improvements to the supermin appliance?
I did some quick tests on what would be possible, see below:
If I try to use virtio 9P instead it becomes a little faster (instead of root fs on block
device):
real 0m3.299s
user 0m2.084s
sys 0m1.828s
To do this we need to load just the virtio and 9p modules:
let kmods = [
"virtio*.ko*";
"9p*.ko*";
]
Modify init.c to mount the 9p root:
https://gist.github.com/anonymous/cb32986fd9b85c315ae4
And wrap qemu to create the 9p device (the mount is just there for a quick test, in
reality supermin itself should probably leave the unpacker dir around):
https://gist.github.com/anonymous/5cdbc18974f88d6ea9e0
Next step: kvmtool [3]. Avoids some delays introduced by legacy BIOS boot, and device
probing, however
lkvm currently lacks support for additional virtio-serial channels, so - although the
supermin appliance would boot - guestfsd wouldn't be able to communicate with the
host:
https://gist.github.com/anonymous/e6b0a12e1da811f24e5b
I tried QBoot [4], which requires that you first strip down your kernel+initrd to fit into
8128k,
and uses an unmodified Qemu otherwise.
But then I haven't been able to get it to actually launch guestfsd, it hangs somewhere
earlier:
https://gist.github.com/anonymous/52fc2890a0884230e4f8
Maybe I removed too many modules here?
Another possibility would be to run guestfsd with rkt+lkvm stage1, or hyper.sh; but I
don't see a way
to forward a unix socket or a virtio-serial channel through them, would guestfsd be able
to work over a TCP socket
or the main virtio console? (its probably preferable to avoid having any networking in the
guestfsd appliance)
[1] https://lwn.net/Articles/644675/
[2] https://hyper.sh/faq.html
[3] https://git.kernel.org/cgit/linux/kernel/git/will/kvmtool.git/tree/README
[4] https://github.com/bonzini/qboot
Best regards,
--
Edwin Török | Co-founder and Lead Developer
Skylable open-source object storage: reliable, fast, secure
http://www.skylable.com
2:34 p.m.
On Sun, Mar 13, 2016 at 04:24:53PM +0200, Török Edwin wrote:
Hi,
I remembered reading about Intel Clear Containers [1], and Hyper/Qboot [2]
that support startup times measured in the hundreds of miliseconds range.
On an AMD FX(tm)-8350 'guestfish -a /dev/null run' takes ~4s when run the 2nd/3rd
time:
real 0m4.152s
user 0m2.120s
sys 0m0.564s
Are there any plans on applying these improvements to the supermin appliance?
I was looking at Clear Containers last week.
I did some quick tests on what would be possible, see below:
If I try to use virtio 9P instead it becomes a little faster (instead of root fs on block
device):
real 0m3.299s
user 0m2.084s
sys 0m1.828s
To do this we need to load just the virtio and 9p modules:
let kmods = [
"virtio*.ko*";
"9p*.ko*";
]
Modify init.c to mount the 9p root:
https://gist.github.com/anonymous/cb32986fd9b85c315ae4
And wrap qemu to create the 9p device (the mount is just there for a quick test, in
reality supermin itself should probably leave the unpacker dir around):
https://gist.github.com/anonymous/5cdbc18974f88d6ea9e0
Next step: kvmtool [3]. Avoids some delays introduced by legacy BIOS boot, and device
probing, however
lkvm currently lacks support for additional virtio-serial channels, so - although the
supermin appliance would boot - guestfsd wouldn't be able to communicate with the
host:
https://gist.github.com/anonymous/e6b0a12e1da811f24e5b
I tried QBoot [4], which requires that you first strip down your kernel+initrd to fit
into 8128k,
and uses an unmodified Qemu otherwise.
But then I haven't been able to get it to actually launch guestfsd, it hangs
somewhere earlier:
https://gist.github.com/anonymous/52fc2890a0884230e4f8
Maybe I removed too many modules here?
Another possibility would be to run guestfsd with rkt+lkvm stage1, or hyper.sh; but I
don't see a way
to forward a unix socket or a virtio-serial channel through them, would guestfsd be able
to work over a TCP socket
or the main virtio console? (its probably preferable to avoid having any networking in
the guestfsd appliance)
[1] https://lwn.net/Articles/644675/
[2] https://hyper.sh/faq.html
[3] https://git.kernel.org/cgit/linux/kernel/git/will/kvmtool.git/tree/README
[4] https://github.com/bonzini/qboot
This is all very good analysis.
The issues that I had in brief were:
(1) We could run kvmtool, perhaps by adding a new backend, but it
seems a better idea to add the required features to qemu. Anything
based on kvmtool wouldn't support qcow2 and the myriad other features
of qemu (see also:
https://rwmj.wordpress.com/2015/11/07/linux-kernel-library-backend-for-li...)
(2) On the kernel side, the Intel kernel contains lots of little
hacks, and many baremetal CONFIG_* options are disabled. Hacks can be
upstreamed once we massage them into shape. The real issue is keeping
a single baremetal + virt kernel image, since separate kernel images
would never fly with Fedora or RHEL. That means we cannot just
disable slow drivers/subsystems by having an alternate kernel with
lots of CONFIG_* options disabled, we need to address those problems
properly.
(3) DAX / NVDIMM / etc - love them. Not supported upstream (either
kernel or qemu) yet.
(4) Passthrough (eg 9p) filesystems. You touched on that above.
Red Hat doesn't much like 9pfs for several reasons, yet we also don't
have a plausible alternative at the moment. This is mainly a problem
for running fast Docker containers, rather than libguestfs though.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
12:44 p.m.
On 03/15/2016 21:34, Richard W.M. Jones wrote:
I was looking at Clear Containers last week.
[...]
This is all very good analysis.
Thanks, looks like I raised the question at a good time :)
The issues that I had in brief were:
(1) We could run kvmtool, perhaps by adding a new backend, but it
seems a better idea to add the required features to qemu. Anything
based on kvmtool wouldn't support qcow2 and the myriad other features
of qemu (see also:
https://rwmj.wordpress.com/2015/11/07/linux-kernel-library-backend-for-li...)
Could qemu-nbd from inside the guest be used? (as a performance/security tradeoff)
(2) On the kernel side, the Intel kernel contains lots of little
hacks, and many baremetal CONFIG_* options are disabled. Hacks can be
upstreamed once we massage them into shape. The real issue is keeping
a single baremetal + virt kernel image, since separate kernel images
would never fly with Fedora or RHEL. That means we cannot just
disable slow drivers/subsystems by having an alternate kernel with
lots of CONFIG_* options disabled, we need to address those problems
properly.
(3) DAX / NVDIMM / etc - love them. Not supported upstream (either
kernel or qemu) yet.
(4) Passthrough (eg 9p) filesystems. You touched on that above.
Red Hat doesn't much like 9pfs for several reasons, yet we also don't
have a plausible alternative at the moment. This is mainly a problem
for running fast Docker containers, rather than libguestfs though.
Another thought: why does guestfish need to boot the appliance more than once?
Could virsh save/restore or managedsave/stop/start be used?
The guestfs appliance seems to be around 80MB saved [*] (perhaps ballooning could help
shrink this, or it could be compressed with lz4/lzop).
[*] I copied the XML and changed some things in it, cause the original failed to save
with: error: Requested operation is not valid: domain is marked for auto destroy
Best regards,
--
Edwin Török | Co-founder and Lead Developer
Skylable open-source object storage: reliable, fast, secure
http://www.skylable.com
1:01 p.m.
On Wed, Mar 16, 2016 at 07:44:54PM +0200, Török Edwin wrote:
On 03/15/2016 21:34, Richard W.M. Jones wrote:
> I was looking at Clear Containers last week.
> [...]
>
> This is all very good analysis.
Thanks, looks like I raised the question at a good time :)
>
> The issues that I had in brief were:
>
> (1) We could run kvmtool, perhaps by adding a new backend, but it
> seems a better idea to add the required features to qemu. Anything
> based on kvmtool wouldn't support qcow2 and the myriad other features
> of qemu (see also:
> https://rwmj.wordpress.com/2015/11/07/linux-kernel-library-backend-for-li...)
Could qemu-nbd from inside the guest be used? (as a
performance/security tradeoff)
There was also a proposed solution for using an [ordinary chroot-
style] container as the libguestfs appliance back in the day. It
involved qemu-nbd (on the host) plus loopback mounting IIRC. I don't
think anyone ever explored it in any depth, since it requires root and
as you say isn't secure.
> (2) On the kernel side, the Intel kernel contains lots of
little
> hacks, and many baremetal CONFIG_* options are disabled. Hacks can be
> upstreamed once we massage them into shape. The real issue is keeping
> a single baremetal + virt kernel image, since separate kernel images
> would never fly with Fedora or RHEL. That means we cannot just
> disable slow drivers/subsystems by having an alternate kernel with
> lots of CONFIG_* options disabled, we need to address those problems
> properly.
>
> (3) DAX / NVDIMM / etc - love them. Not supported upstream (either
> kernel or qemu) yet.
>
> (4) Passthrough (eg 9p) filesystems. You touched on that above.
> Red Hat doesn't much like 9pfs for several reasons, yet we also don't
> have a plausible alternative at the moment. This is mainly a problem
> for running fast Docker containers, rather than libguestfs though.
Another thought: why does guestfish need to boot the appliance more
than once? Could virsh save/restore or managedsave/stop/start be
used?
This was once proposed too. It requires a complete change to the way
we build and manage the appliance, and it was unclear if there would
be much benefit.
The guestfs appliance seems to be around 80MB saved [*] (perhaps
ballooning could help shrink this, or it could be compressed with lz4/lzop).
[*] I copied the XML and changed some things in it, cause the original failed to save
with: error: Requested operation is not valid: domain is marked for auto destroy
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
3207
days inactive
3210
days old
3 comments
2 participants
participants (2)
-
Richard W.M. Jones -
Török Edwin