[Libguestfs] Re: Setting up guestfs mirrors - v2

+ cc kdevops and Ross from the debian side. On Fri, Mar 29, 2024 at 09:17:52AM +0000, Richard W.M. Jones wrote: > > On Thu, Mar 28, 2024 at 09:33:50AM -0700, Luis Chamberlain wrote: > > > [ v2 as I don't think the v1 email got through to guestfs list ] > > > > > > Thanks to the work by Jeff Layton, we've embraced guestfs on kdevops > > > [0] as a replacement to vagrant. To save bandwidth kdevops lets you > > > set up mirrors for all git trees we need, it would be nice to make it > > > easier for folks to also setup mirrors for guestfs base images, the > > > base images are here: > > > > > > https://builder.libguestfs.org/ > > > > > > I take it we'd just add say > > > $XDG_CONFIG_HOME/virt-builder/repos.d/kdevops-local-mirror.conf with > > > the new URL where we established the mirror, is that correct? > > Yes. Neat! > > > I was looking to see if there was an rsync mirror for the main URL > > > where all the images are above, or is there a better way to get the > > > latest updates? > > Ideally we'd like to not host the images at all, but getting distros > to host them instead has not been very successful (except for SUSE). It makes sense, it would put all the burden on just one server, and the licensing considerations are already figured out by distros as well so might as well leverage that. So I see four for distros to host them: 1) decentralizes 2) distros already produce cloud images, having a guestfs image should be trivial to produce, just as current vagrant images 3) licensing is already figured out 4) since some distros also provie updated images than the base, as with debian testing vagrant images, they can also provide more up to date guestfs images as well, reducing the amount of time it takes to have an up to date system. If distros do start shipping guestfs images, then eventually we want to update upstream virt-builder with a respective: $XDG_CONFIG_HOME/virt-builder/repos.d/$distro.conf That would allo $distro's source spelled out for their own updates. A good example as you hinted was that opensuse already does exactly this, so we have: cat /etc/virt-builder/repos.d/opensuse.conf [opensuse.org] uri=http://download.opensuse.org/repositories/Virtualization:/virt-builde... gpgkey=file:///etc/virt-builder/repos.d/opensuse.gpg > > > Also, to save even more time with images, it would be useful if we had > > > base images getting updates a bit more regularly so that the step to > > > apt-get upgrade after bringup won't take too long by using the latest > > > base images. It seems this is possible with customer images, but it > > > would be nicer if we had this as a general thing. Any ideas to support > > > that? > > Yes, we'd like Debian to generate their own images! Actually > they don't even need to generate images at all, just host > the index file. Great! Let's all start helping to persuade each distro to do this then. In the meantime, to help alleviate the load off of builder.libguestfs.org it would be good to at laest host a few mirrors, maybe some public maybe some not public. To help with this I'd like some clarifications. I just discovered and ran virt-builder --cache-all-templates, which downloads all the 29 GiB of base images onto my ~/.cache/virt-builder/ but I don't see the index file there, so this does not seem to be an equivalent of what would be an rsync of the data on https://builder.libguestfs.org/ so I don't think the contents of this directory could be used without postprocessing of the same data on the builder page.

This is not an issue if all one wants is to just mirror the data locally, but say if one wanted to *host* a exact mirror of what you have in the URL it would need some post processing. For instance the above URL has a XZ compressed debian-12.xz while the command virt-builder --cache-all-templates gets me an XZ compressed file ~/.cache/virt-builder/debian-12.x86_64.1 decompressing both and sha1suming confirms its the same file with an sha1sum of 6fc19d34398c2c57ebd396c080d63531238f3ac43b61c66ae167903188404bf44de29b677085b9419f906a30cc720afc3fe43af7e09d970beadd8b673a06ca86 and with it compressed as reflected in the public index file: f1c58b53b7d25691ccbd2c3eb97f692985e8a035247dee3ed8e024bbae64574620ace75dbb57207064739f5e4b317c6ad6d6d518d777c719441b06b1634fe07e Is there a better way to host an exact copy of what's in https://builder.libguestfs.org and what would a conf that does this look like?

Let's say we want to implement private or public mirrors to help, once we figure out how to mirror mirror same data, and we dump it on page say on foobar.org would we want to add to a system: /etc/virt-builder/repos.d/foobar.conf [foobar.org] uri=http://foobar.org/download/builder/index.asc gpgkey=file:///etc/xdg/virt-builder/repos.d/libguestfs.gpg