Re: [Libguestfs] nbdkit / exposing disk images in containers

On Sat, Jul 11, 2020 at 11:18 AM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: > > KubeVirt is a custom resource (a kind of plugin) for Kubernetes which > adds support for running virtual machines. As part of this they have > the same problems as everyone else of how to import large disk images > into the system for pets, templates, etc. > > As part of the project they've defined a format for embedding a disk > image into a container (unclear why? perhaps so these can be > distributed using the existing container registry systems?): > > https://github.com/kubevirt/containerized-data-importer/blob/master/doc/i... > > An example of such a disk-in-a-container is here: > > https://hub.docker.com/r/kubevirt/fedora-cloud-container-disk-demo > > We've been asked if we can help with tools to efficiently import these > disk images, and I have suggested a few things with nbdkit and have > written a couple of filters (tar, gzip) to support this. I don't think gzip filter matches nbdkit very well. Having to decompress the entire disk before you can serve it does not sound right.

> This email is my thoughts on further development work in this area. > > ---------------------------------------------------------------------- > > (1) Accessing the disk image directly from the Docker Hub. > > When you get down to it, what this actually is: > > * There is a disk image in qcow2 format. > > * It is embedded as "./disk/downloaded" in a gzip-compressed tar > file. (This is a container with a single layer). > > * This tarball is uploaded to (in this case) the Docker Hub and can > be accessed over a URL. The URL can be constructed using a few > json requests. > > * The URL is served by nginx and this supports HTTP range requests. > > I encapsulated all of this in the attached script. This is an > existence proof that it is possible to access the image with nbdkit. > > One problem is that the auth token only lasts for a limited time > (seems to be 5 minutes in my test), and it doesn't automatically renew > as you download the layer, so if the download takes longer than 5 > minutes you'll suddenly get unrecoverable authorization failures. > > There seem to be two possible ways to solve this: > > (a) Write a new nbdkit-container-plugin which does the authorization > (essentially hiding most of the details in the attached script > from the user). It could deal with renewing the key as > required. > > (b) Modify nbdkit-curl-plugin so the user could provide a script for > renewing authorization. This would expose the full gory details > to the end user, but on the other hand might be useful in other > situations that require authorization. docker/podman already solved this, why should nbdkit solve it again?

Do you get timeouts while you download the image with a single request?

> (2) nbdkit-tar-filter exportname and listing files. > > This has already been covered by an email from Nir Soffer, so I'll > simply link to that: > > https://lists.gnu.org/archive/html/qemu-discuss/2020-06/msg00058.html > > It basically requires a fairly simple change to nbdkit-tar-filter to > map the tar filenames into export names, and a deeper change to nbdkit > core server to allow listing all export names. The end result would > be that an NBD client could query the list of files [ie exports] in > the tarball and choose one to download. We know the tar member name upfront, so why do we need to list the contents?

> (3) gzip & tar require full downloads - why not “docker/podman save/export”? This looks like a better direction. The nice thing about embedding the disk in the container image is being able to use existing infrastructure (docker, quay) to host the images, and to transfer them to the hosts. We don't need to write any code for this. Even better, we have automatic caching on the host by docker/podman, so we have to pull the image from the registry only once on every host. Then we can access the local cache. > Stepping back to get the bigger picture: Because the OCI standard uses > gzip for compression (https://stackoverflow.com/a/9213826), and > because the tar index is interspersed with the tar data, you always > need to download the whole container layer before you can access the > disk image inside. You need to download most of the tar, but you don't need to keep the tar in a temporary file. For example in python you can create a tarfile over with the http response object in streaming with transparent decompression mode ("r|*"), and stream the disk contents from the tar without a temporary file. with tarfile.open(mode="r|*", fileobj=response) as tar: for member in tar: if member.name == "./disk/downloaded": with tar.extractfile(member) as f shutil.copyfileobj(f, sys.stdout.buffer) sys.exit(0) I think this is what cdi import code does, and is the most efficient way to copy the disk directly from the registry with the current format. > Currently nbdkit-gzip-filter hides this from the > end user, but it's still downloading the whole thing to a temporary > file. There's no way round that unless OCI can be persuaded to use a > better format. The way is to use the container image downloaded by podman/docker. > But docker/podman already has a way to export container layers, > ie. the save and export commands. These also have the advantage that > it will cache the downloaded layers between runs. So why aren't we > using that? > > In this world, nbdkit-container-plugin would simply use docker/podman > save (or export?) to grab the container as a tar file, and we would > use the tar filter as above to expose the contents as an NBD endpoint > for further consumption. IOW: > > nbdkit container docker.io/kubevirt/fedora-cloud-container-disk-demo \ > --filter=tar tar-entry=./downloaded/disk This will work but there are 2 issues: 1. podman save/export copy the tar locally. This is pretty fast for the example image but copying the tar and deleting it seems wasteful. 2. If we have the tar locally, why not use qemu-img directly? we can find the offset of the disk inside the tar and use: $ time podman save --format oci-dir -o demo-oci docker.io/kubevirt/fedora-cloud-container-disk-demo real 0m2.795s user 0m2.011s sys 0m0.878s $ time qemu-img convert -O raw 'json:{"file": {"driver": "raw", "offset": 1536, "file": {"driver": "file", "filename": "demo-oci/8162f3eda33d5a87df56e969dcd9777523bd53278a0701b2e53b93c33c01853e"}}}' out.raw real 0m1.036s user 0m3.237s sys 0m1.326s But I think we have a better way - using a self-extracting-disk container. Start a container with a disk image, and run qemu-img inside this container to convert the disk to the target PV. It can work like this: 1. We create a base image - this will be used for all disks container images. $ cat Dockerfile.kubevirt-img FROM alpine RUN apk add qemu-img $ podman build -t kubevirt-img -f Dockerfile.kubevirt-img . ... You pull this from quay.io/nirsof/kubevirt-img. 2. Create a disk container image, based on the base image $ cat Dockerfile.kubevirt-fedora-cloud-disk FROM quay.io/nirsof/kubevirt-fimg COPY disk.qcow2 /disk.qcow2 CMD ["qemu-img", "convert", "-p", "-f", "qcow2", "-O", "raw", "/disk.qcow2", "/target/disk.img"] $ podman build -t kubevirt-fedora-cloud-disk -f Dockerfile.kubevirt-fedora-cloud-disk . ... This container is a little larger, but the common layer with qemu-img and its dependencies is shared between all disk container images. In this example it adds only 25 MiB. You can pull this from quay.io/nirsof/kubevirt-fedora-cloud-disk. With this we can create a copy of the disk using: $ time podman run --volume ./:/target:Z --rm -it quay.io/nirsof/kubevirt-fedora-cloud-disk Trying to pull quay.io/nirsof/kubevirt-fedora-cloud-disk... Getting image source signatures Copying blob 0d9094d70e9c skipped: already exists Copying blob a3ed95caeb02 done Copying blob a3ed95caeb02 done Copying blob 18717781bd09 done Copying blob fe5cd0d8bf32 done Writing manifest to image destination Storing signatures (100.00/100%) real 0m59.800s user 0m8.988s sys 0m7.437s $ ls -lhs disk.img 728M -rw-r--r--. 1 nsoffer nsoffer 4.0G Jul 12 21:45 disk.img $ podman images | grep fedora-cloud quay.io/nirsof/kubevirt-fedora-cloud-disk latest 097ef06b6d71 About an hour ago 326 MB docker.io/kubevirt/fedora-cloud-container-disk-demo latest 6494830c6dc7 50 years ago 303 MB The next time we run this we get the container from the cache: $ time podman run --volume ./:/target:Z --rm -it quay.io/nirsof/kubevirt-fedora-cloud-disk (100.00/100%) real 0m2.244s user 0m0.070s sys 0m0.253s