3:16 p.m.
On Sat, Jul 11, 2020 at 11:18 AM Richard W.M. Jones <rjones(a)redhat.com> wrote:
KubeVirt is a custom resource (a kind of plugin) for Kubernetes which
adds support for running virtual machines. As part of this they have
the same problems as everyone else of how to import large disk images
into the system for pets, templates, etc.
As part of the project they've defined a format for embedding a disk
image into a container (unclear why? perhaps so these can be
distributed using the existing container registry systems?):
https://github.com/kubevirt/containerized-data-importer/blob/master/doc/i...
An example of such a disk-in-a-container is here:
https://hub.docker.com/r/kubevirt/fedora-cloud-container-disk-demo
We've been asked if we can help with tools to efficiently import these
disk images, and I have suggested a few things with nbdkit and have
written a couple of filters (tar, gzip) to support this.
I don't think gzip filter matches nbdkit very well. Having to decompress the
entire disk before you can serve it does not sound right.
This email is my thoughts on further development work in this area.
----------------------------------------------------------------------
(1) Accessing the disk image directly from the Docker Hub.
When you get down to it, what this actually is:
* There is a disk image in qcow2 format.
* It is embedded as "./disk/downloaded" in a gzip-compressed tar
file. (This is a container with a single layer).
* This tarball is uploaded to (in this case) the Docker Hub and can
be accessed over a URL. The URL can be constructed using a few
json requests.
* The URL is served by nginx and this supports HTTP range requests.
I encapsulated all of this in the attached script. This is an
existence proof that it is possible to access the image with nbdkit.
One problem is that the auth token only lasts for a limited time
(seems to be 5 minutes in my test), and it doesn't automatically renew
as you download the layer, so if the download takes longer than 5
minutes you'll suddenly get unrecoverable authorization failures.
There seem to be two possible ways to solve this:
(a) Write a new nbdkit-container-plugin which does the authorization
(essentially hiding most of the details in the attached script
from the user). It could deal with renewing the key as
required.
(b) Modify nbdkit-curl-plugin so the user could provide a script for
renewing authorization. This would expose the full gory details
to the end user, but on the other hand might be useful in other
situations that require authorization.
docker/podman already solved this, why should nbdkit solve it again?
Do you get timeouts while you download the image with a single request?
(2) nbdkit-tar-filter exportname and listing files.
This has already been covered by an email from Nir Soffer, so I'll
simply link to that:
https://lists.gnu.org/archive/html/qemu-discuss/2020-06/msg00058.html
It basically requires a fairly simple change to nbdkit-tar-filter to
map the tar filenames into export names, and a deeper change to nbdkit
core server to allow listing all export names. The end result would
be that an NBD client could query the list of files [ie exports] in
the tarball and choose one to download.
We know the tar member name upfront, so why do we need to list the contents?
(3) gzip & tar require full downloads - why not “docker/podman
save/export”?
This looks like a better direction.
The nice thing about embedding the disk in the container image is being able
to use existing infrastructure (docker, quay) to host the images, and
to transfer
them to the hosts. We don't need to write any code for this.
Even better, we have automatic caching on the host by docker/podman, so we
have to pull the image from the registry only once on every host. Then we can
access the local cache.
Stepping back to get the bigger picture: Because the OCI standard
uses
gzip for compression (https://stackoverflow.com/a/9213826), and
because the tar index is interspersed with the tar data, you always
need to download the whole container layer before you can access the
disk image inside.
You need to download most of the tar, but you don't need to keep the tar
in a temporary file. For example in python you can create a tarfile over with
the http response object in streaming with transparent decompression mode
("r|*"), and stream the disk contents from the tar without a temporary file.
with tarfile.open(mode="r|*", fileobj=response) as tar:
for member in tar:
if member.name == "./disk/downloaded":
with tar.extractfile(member) as f
shutil.copyfileobj(f, sys.stdout.buffer)
sys.exit(0)
I think this is what cdi import code does, and is the most efficient way
to copy the disk directly from the registry with the current format.
Currently nbdkit-gzip-filter hides this from the
end user, but it's still downloading the whole thing to a temporary
file. There's no way round that unless OCI can be persuaded to use a
better format.
The way is to use the container image downloaded by podman/docker.
But docker/podman already has a way to export container layers,
ie. the save and export commands. These also have the advantage that
it will cache the downloaded layers between runs. So why aren't we
using that?
In this world, nbdkit-container-plugin would simply use docker/podman
save (or export?) to grab the container as a tar file, and we would
use the tar filter as above to expose the contents as an NBD endpoint
for further consumption. IOW:
nbdkit container docker.io/kubevirt/fedora-cloud-container-disk-demo \
--filter=tar tar-entry=./downloaded/disk
This will work but there are 2 issues:
1. podman save/export copy the tar locally. This is pretty fast for the example
image but copying the tar and deleting it seems wasteful.
2. If we have the tar locally, why not use qemu-img directly? we can find the
offset of the disk inside the tar and use:
$ time podman save --format oci-dir -o demo-oci
docker.io/kubevirt/fedora-cloud-container-disk-demo
real 0m2.795s
user 0m2.011s
sys 0m0.878s
$ time qemu-img convert -O raw 'json:{"file": {"driver":
"raw",
"offset": 1536, "file": {"driver": "file",
"filename":
"demo-oci/8162f3eda33d5a87df56e969dcd9777523bd53278a0701b2e53b93c33c01853e"}}}'
out.raw
real 0m1.036s
user 0m3.237s
sys 0m1.326s
But I think we have a better way - using a self-extracting-disk
container. Start a container with
a disk image, and run qemu-img inside this container to convert the
disk to the target PV.
It can work like this:
1. We create a base image - this will be used for all disks container images.
$ cat Dockerfile.kubevirt-img
FROM alpine
RUN apk add qemu-img
$ podman build -t kubevirt-img -f Dockerfile.kubevirt-img .
...
You pull this from quay.io/nirsof/kubevirt-img.
2. Create a disk container image, based on the base image
$ cat Dockerfile.kubevirt-fedora-cloud-disk
FROM quay.io/nirsof/kubevirt-fimg
COPY disk.qcow2 /disk.qcow2
CMD ["qemu-img", "convert", "-p", "-f",
"qcow2", "-O", "raw",
"/disk.qcow2", "/target/disk.img"]
$ podman build -t kubevirt-fedora-cloud-disk -f
Dockerfile.kubevirt-fedora-cloud-disk .
...
This container is a little larger, but the common layer with qemu-img
and its dependencies is
shared between all disk container images. In this example it adds only 25 MiB.
You can pull this from quay.io/nirsof/kubevirt-fedora-cloud-disk.
With this we can create a copy of the disk using:
$ time podman run --volume ./:/target:Z --rm -it
quay.io/nirsof/kubevirt-fedora-cloud-disk
Trying to pull quay.io/nirsof/kubevirt-fedora-cloud-disk...
Getting image source signatures
Copying blob 0d9094d70e9c skipped: already exists
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 done
Copying blob 18717781bd09 done
Copying blob fe5cd0d8bf32 done
Writing manifest to image destination
Storing signatures
(100.00/100%)
real 0m59.800s
user 0m8.988s
sys 0m7.437s
$ ls -lhs disk.img
728M -rw-r--r--. 1 nsoffer nsoffer 4.0G Jul 12 21:45 disk.img
$ podman images | grep fedora-cloud
quay.io/nirsof/kubevirt-fedora-cloud-disk latest
097ef06b6d71 About an hour ago 326 MB
docker.io/kubevirt/fedora-cloud-container-disk-demo latest
6494830c6dc7 50 years ago 303 MB
The next time we run this we get the container from the cache:
$ time podman run --volume ./:/target:Z --rm -it
quay.io/nirsof/kubevirt-fedora-cloud-disk
(100.00/100%)
real 0m2.244s
user 0m0.070s
sys 0m0.253s
Nir