September 2021 - Libguestfs - libguestfs List Archives

[libnbd PATCH] ci: Adopt libvirt-ci's manifest approach
by Martin Kletzander 23 Sep '21

23 Sep '21

This makes it a bit easier to write .gitlab-ci.yml as it inherits most of what can be generated from the ci/gitlab.yml and deprecates the ci/refresh script. Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com> --- .gitlab-ci.yml | 270 +----------- ci/README.rst | 7 +- ci/cirrus/freebsd-12.vars | 2 +- ci/cirrus/freebsd-13.vars | 2 +- ci/cirrus/freebsd-current.vars | 2 +- ci/cirrus/macos-11.vars | 2 +- ci/containers/alpine-314.Dockerfile | 2 +- ci/containers/alpine-edge.Dockerfile | 2 +- ci/containers/centos-8.Dockerfile | 2 +- ci/containers/centos-stream-8.Dockerfile | 2 +- ci/containers/debian-10-cross-i686.Dockerfile | 2 +- ci/containers/debian-10.Dockerfile | 2 +- ci/containers/debian-11-cross-i686.Dockerfile | 2 +- ci/containers/debian-11.Dockerfile | 2 +- ci/containers/debian-sid.Dockerfile | 2 +- ci/containers/fedora-33.Dockerfile | 2 +- ci/containers/fedora-34-cross-i686.Dockerfile | 0 ci/containers/fedora-34.Dockerfile | 2 +- ci/containers/fedora-rawhide.Dockerfile | 2 +- ci/containers/opensuse-leap-152.Dockerfile | 2 +- ci/containers/opensuse-tumbleweed.Dockerfile | 2 +- ci/containers/ubuntu-1804.Dockerfile | 2 +- ci/containers/ubuntu-2004.Dockerfile | 2 +- ci/gitlab.yml | 413 ++++++++++++++++++ ci/manifest.yml | 72 +++ ci/refresh | 33 -- 26 files changed, 520 insertions(+), 315 deletions(-) delete mode 100644 ci/containers/fedora-34-cross-i686.Dockerfile create mode 100644 ci/gitlab.yml create mode 100644 ci/manifest.yml delete mode 100755 ci/refresh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 8976015b1e20..122bc0860178 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -13,30 +13,10 @@ stages: export CCACHE_MAXSIZE="500M" export PATH="$CCACHE_WRAPPERSDIR:$PATH" -# Common templates - -.container_job: - image: docker:stable - stage: containers - needs: [] - services: - - name: registry.gitlab.com/libvirt/libvirt-ci/docker-dind:master - alias: docker - before_script: - - export TAG="$CI_REGISTRY_IMAGE/$NAME:latest" - - export COMMON_TAG="$CI_REGISTRY/nbdkit/libnbd/$NAME:latest" - - docker info - - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" - script: - - docker pull "$TAG" || docker pull "$COMMON_TAG" || true - - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/containers/$NAME.Dockerfile" ci/containers - - docker push "$TAG" - after_script: - - docker logout +include: '/ci/gitlab.yml' .base_build_job: stage: builds - image: $CI_REGISTRY_IMAGE/$NAME:latest cache: paths: - ccache/ @@ -52,6 +32,16 @@ stages: public: true when: always +.native_build_job: + extends: + - .base_build_job + - .gitlab_native_build_job + +.cross_build_job: + extends: + - .base_build_job + - .gitlab_cross_build_job + # Jobs that we delegate to Cirrus CI because they require an operating # system other than Linux. These jobs will only run if the required # setup has been performed on the GitLab account (see ci/README.rst). @@ -91,241 +81,3 @@ stages: - cirrus-run -v --show-build-log always ci/cirrus/$NAME.yml rules: - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN" - -# Cross-compilation container build jobs -i686-debian-10-container: - extends: .container_job - variables: - NAME: debian-10-cross-i686 - -i686-debian-11-container: - extends: .container_job - variables: - NAME: debian-11-cross-i686 - -# Native container build jobs -x64-alpine-314-container: - extends: .container_job - variables: - NAME: alpine-314 - -x64-alpine-edge-container: - extends: .container_job - variables: - NAME: alpine-edge - -x64-centos-8-container: - extends: .container_job - variables: - NAME: centos-8 - -x64-centos-stream-8-container: - extends: .container_job - variables: - NAME: centos-stream-8 - -x64-debian-11-container: - extends: .container_job - variables: - NAME: debian-11 - -x64-debian-sid-container: - extends: .container_job - variables: - NAME: debian-sid - -x64-fedora-33-container: - extends: .container_job - variables: - NAME: fedora-33 - -x64-fedora-34-container: - extends: .container_job - variables: - NAME: fedora-34 - -x64-fedora-rawhide-container: - extends: .container_job - allow_failure: true - variables: - NAME: fedora-rawhide - -x64-opensuse-leap-152-container: - extends: .container_job - variables: - NAME: opensuse-leap-152 - -x64-opensuse-tumbleweed-container: - extends: .container_job - variables: - NAME: opensuse-tumbleweed - -x64-ubuntu-1804-container: - extends: .container_job - variables: - NAME: ubuntu-1804 - -x64-ubuntu-2004-container: - extends: .container_job - variables: - NAME: ubuntu-2004 - -# Cross-compilation build jobs -i686-debian-10: - extends: .base_build_job - needs: - - i686-debian-10-container - variables: - NAME: debian-10-cross-i686 - CROSS: i686 - -i686-debian-11: - extends: .base_build_job - needs: - - i686-debian-11-container - variables: - NAME: debian-11-cross-i686 - CROSS: i686 - -# Native architecture build + test jobs -x64-debian-11: - extends: .base_build_job - needs: - - x64-debian-11-container - variables: - NAME: debian-11 - -x64-debian-sid: - extends: .base_build_job - needs: - - x64-debian-sid-container - variables: - NAME: debian-sid - -x64-centos-8: - extends: .base_build_job - needs: - - x64-centos-8-container - variables: - NAME: centos-8 - -x64-alpine-314: - extends: .base_build_job - needs: - - x64-alpine-314-container - variables: - NAME: alpine-314 - -x64-alpine-edge: - extends: .base_build_job - needs: - - x64-alpine-edge-container - variables: - NAME: alpine-edge - -x64-centos-8-clang: - extends: .base_build_job - needs: - - x64-centos-8-container - variables: - NAME: centos-8 - CC: clang - -x64-centos-stream-8: - extends: .base_build_job - needs: - - x64-centos-stream-8-container - variables: - NAME: centos-stream-8 - -x64-fedora-33: - extends: .base_build_job - needs: - - x64-fedora-33-container - variables: - NAME: fedora-33 - -x64-fedora-34: - extends: .base_build_job - needs: - - x64-fedora-34-container - variables: - NAME: fedora-34 - -x64-fedora-rawhide: - extends: .base_build_job - needs: - - x64-fedora-rawhide-container - variables: - NAME: fedora-rawhide - -x64-fedora-rawhide-clang: - extends: .base_build_job - needs: - - x64-fedora-rawhide-container - variables: - NAME: fedora-rawhide - CC: clang - -x64-opensuse-leap-152: - extends: .base_build_job - needs: - - x64-opensuse-leap-152-container - variables: - NAME: opensuse-leap-152 - -x64-opensuse-tumbleweed: - extends: .base_build_job - needs: - - x64-opensuse-tumbleweed-container - variables: - NAME: opensuse-tumbleweed - -x64-ubuntu-1804: - extends: .base_build_job - needs: - - x64-ubuntu-1804-container - variables: - NAME: ubuntu-1804 - GOLANG: skip - -x64-ubuntu-2004: - extends: .base_build_job - needs: - - x64-ubuntu-2004-container - variables: - NAME: ubuntu-2004 - -x64-freebsd-12-build: - extends: .cirrus_build_job - variables: - NAME: freebsd-12 - CIRRUS_VM_INSTANCE_TYPE: freebsd_instance - CIRRUS_VM_IMAGE_SELECTOR: image_family - CIRRUS_VM_IMAGE_NAME: freebsd-12-2 - UPDATE_COMMAND: pkg update - INSTALL_COMMAND: pkg install -y - -x64-freebsd-13-build: - extends: .cirrus_build_job - variables: - NAME: freebsd-13 - CIRRUS_VM_INSTANCE_TYPE: freebsd_instance - CIRRUS_VM_IMAGE_SELECTOR: image_family - CIRRUS_VM_IMAGE_NAME: freebsd-13-0 - UPDATE_COMMAND: pkg update - INSTALL_COMMAND: pkg install -y - -x64-macos-11-build: - extends: .cirrus_build_job - allow_failure: true - variables: - NAME: macos-11 - CIRRUS_VM_INSTANCE_TYPE: osx_instance - CIRRUS_VM_IMAGE_SELECTOR: image - CIRRUS_VM_IMAGE_NAME: big-sur-base - UPDATE_COMMAND: brew update - UPGRADE_COMMAND: brew upgrade - INSTALL_COMMAND: brew install - PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin - PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig diff --git a/ci/README.rst b/ci/README.rst index c1c97362e41f..3d11e3a4d8bc 100644 --- a/ci/README.rst +++ b/ci/README.rst @@ -35,9 +35,10 @@ Container images are prepared during the build and cached, so that they do not need to be rebuilt for every run, or even built from scratch when changed slightly. The cached container images can be deleted at any time as they will be correctly rebuilt in case of a cache miss. The Dockerfiles are generated -using ``lcitool`` and stored under ``ci/containers``. In that directory is also -a ``refresh`` script which can regenerate those files, for example after an -update to libvirt-ci (e.g. when adding a new dependency). +using ``lcitool`` and stored under ``ci/containers``. In order to update the +files one needs to run ``lcitool manifest ci/manifest.yml`` from the root of the +git repository, for example after an update to libvirt-ci (e.g. when adding a +new dependency). Recreating the builds locally is pretty straightforward. Choose a file from ``ci/containers`` which represents the desired setup. In this example let's diff --git a/ci/cirrus/freebsd-12.vars b/ci/cirrus/freebsd-12.vars index 2573de241849..6f2b76ce5d11 100644 --- a/ci/cirrus/freebsd-12.vars +++ b/ci/cirrus/freebsd-12.vars @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool variables freebsd-12 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/cirrus/freebsd-13.vars b/ci/cirrus/freebsd-13.vars index 40e8ddd43e49..6f2b76ce5d11 100644 --- a/ci/cirrus/freebsd-13.vars +++ b/ci/cirrus/freebsd-13.vars @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool variables freebsd-13 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/cirrus/freebsd-current.vars b/ci/cirrus/freebsd-current.vars index 73dff989ffb4..6f2b76ce5d11 100644 --- a/ci/cirrus/freebsd-current.vars +++ b/ci/cirrus/freebsd-current.vars @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool variables freebsd-current libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/cirrus/macos-11.vars b/ci/cirrus/macos-11.vars index eae75535a960..89bda78fd154 100644 --- a/ci/cirrus/macos-11.vars +++ b/ci/cirrus/macos-11.vars @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool variables macos-11 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/alpine-314.Dockerfile b/ci/containers/alpine-314.Dockerfile index 0fb954abb66b..82960f71ecff 100644 --- a/ci/containers/alpine-314.Dockerfile +++ b/ci/containers/alpine-314.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile alpine-314 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/alpine-edge.Dockerfile b/ci/containers/alpine-edge.Dockerfile index e27775d647fb..3e9fbfa223eb 100644 --- a/ci/containers/alpine-edge.Dockerfile +++ b/ci/containers/alpine-edge.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile alpine-edge libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/centos-8.Dockerfile b/ci/containers/centos-8.Dockerfile index 1e05d4cda98a..d46b428f726c 100644 --- a/ci/containers/centos-8.Dockerfile +++ b/ci/containers/centos-8.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile centos-8 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/centos-stream-8.Dockerfile b/ci/containers/centos-stream-8.Dockerfile index 8f2dff57b3a3..613d05d6ec0c 100644 --- a/ci/containers/centos-stream-8.Dockerfile +++ b/ci/containers/centos-stream-8.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile centos-stream-8 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/debian-10-cross-i686.Dockerfile b/ci/containers/debian-10-cross-i686.Dockerfile index c7c8dca8df4e..5641f58b71fe 100644 --- a/ci/containers/debian-10-cross-i686.Dockerfile +++ b/ci/containers/debian-10-cross-i686.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile --cross i686 debian-10 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/debian-10.Dockerfile b/ci/containers/debian-10.Dockerfile index fbea28350a06..6cc51ec44de2 100644 --- a/ci/containers/debian-10.Dockerfile +++ b/ci/containers/debian-10.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile debian-10 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/debian-11-cross-i686.Dockerfile b/ci/containers/debian-11-cross-i686.Dockerfile index e0d4fc15961d..fe19aa21c2ed 100644 --- a/ci/containers/debian-11-cross-i686.Dockerfile +++ b/ci/containers/debian-11-cross-i686.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile --cross i686 debian-11 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/debian-11.Dockerfile b/ci/containers/debian-11.Dockerfile index c817870daa73..cb52e7eb4db3 100644 --- a/ci/containers/debian-11.Dockerfile +++ b/ci/containers/debian-11.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile debian-11 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/debian-sid.Dockerfile b/ci/containers/debian-sid.Dockerfile index 909b772c473f..e5a3b663b105 100644 --- a/ci/containers/debian-sid.Dockerfile +++ b/ci/containers/debian-sid.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile debian-sid libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/fedora-33.Dockerfile b/ci/containers/fedora-33.Dockerfile index 353aa0d1e789..9c0bfa954182 100644 --- a/ci/containers/fedora-33.Dockerfile +++ b/ci/containers/fedora-33.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile fedora-33 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/fedora-34-cross-i686.Dockerfile b/ci/containers/fedora-34-cross-i686.Dockerfile deleted file mode 100644 index e69de29bb2d1..000000000000 diff --git a/ci/containers/fedora-34.Dockerfile b/ci/containers/fedora-34.Dockerfile index 35d84cc3e0f2..b3f57abfd096 100644 --- a/ci/containers/fedora-34.Dockerfile +++ b/ci/containers/fedora-34.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile fedora-34 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/fedora-rawhide.Dockerfile b/ci/containers/fedora-rawhide.Dockerfile index acbb61f0e74a..49c3e69cc7d6 100644 --- a/ci/containers/fedora-rawhide.Dockerfile +++ b/ci/containers/fedora-rawhide.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile fedora-rawhide libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/opensuse-leap-152.Dockerfile b/ci/containers/opensuse-leap-152.Dockerfile index 050ec442151e..330399253786 100644 --- a/ci/containers/opensuse-leap-152.Dockerfile +++ b/ci/containers/opensuse-leap-152.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile opensuse-leap-152 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/opensuse-tumbleweed.Dockerfile b/ci/containers/opensuse-tumbleweed.Dockerfile index bbcdd3220094..96d4298a5db1 100644 --- a/ci/containers/opensuse-tumbleweed.Dockerfile +++ b/ci/containers/opensuse-tumbleweed.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile opensuse-tumbleweed libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/ubuntu-1804.Dockerfile b/ci/containers/ubuntu-1804.Dockerfile index 72dc53b39c13..a172a0cbac4e 100644 --- a/ci/containers/ubuntu-1804.Dockerfile +++ b/ci/containers/ubuntu-1804.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile ubuntu-1804 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/containers/ubuntu-2004.Dockerfile b/ci/containers/ubuntu-2004.Dockerfile index 6a11fae8d27c..88ea6cfe70dd 100644 --- a/ci/containers/ubuntu-2004.Dockerfile +++ b/ci/containers/ubuntu-2004.Dockerfile @@ -1,6 +1,6 @@ # THIS FILE WAS AUTO-GENERATED # -# $ lcitool dockerfile ubuntu-2004 libnbd +# $ lcitool manifest ci/manifest.yml # # https://gitlab.com/libvirt/libvirt-ci diff --git a/ci/gitlab.yml b/ci/gitlab.yml new file mode 100644 index 000000000000..df3f68ad5587 --- /dev/null +++ b/ci/gitlab.yml @@ -0,0 +1,413 @@ +# THIS FILE WAS AUTO-GENERATED +# +# $ lcitool manifest ci/manifest.yml +# +# https://gitlab.com/libvirt/libvirt-ci + + +.container_job: + image: docker:stable + stage: containers + needs: [] + services: + - name: registry.gitlab.com/libvirt/libvirt-ci/docker-dind:master + alias: docker + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/nbdkit/libnbd/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/containers/$NAME.Dockerfile" ci/containers + - docker push "$TAG" + after_script: + - docker logout + + +.gitlab_native_build_job: + image: $CI_REGISTRY_IMAGE/ci-$NAME:latest + stage: builds + + +.gitlab_cross_build_job: + image: $CI_REGISTRY_IMAGE/ci-$NAME-cross-$CROSS:latest + stage: builds + + +.cirrus_build_job: + stage: builds + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master + needs: [] + script: + - source ci/cirrus/$NAME.vars + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g" + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g" + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g" + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g" + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g" + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g" + -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g" + -e "s|[@]UPGRADE_COMMAND@|$UPGRADE_COMMAND|g" + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g" + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g" + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g" + -e "s|[@]PKGS@|$PKGS|g" + -e "s|[@]MAKE@|$MAKE|g" + -e "s|[@]PYTHON@|$PYTHON|g" + -e "s|[@]PIP3@|$PIP3|g" + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" + -e "s|[@]XML_CATALOG_FILES@|$XML_CATALOG_FILES|g" + <ci/cirrus/build.yml >ci/cirrus/$NAME.yml + - cat ci/cirrus/$NAME.yml + - cirrus-run -v --show-build-log always ci/cirrus/$NAME.yml + rules: + - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN" + + +# Native container jobs + +x86_64-alpine-314-container: + extends: .container_job + allow_failure: false + variables: + NAME: alpine-314 + + +x86_64-alpine-edge-container: + extends: .container_job + allow_failure: false + variables: + NAME: alpine-edge + + +x86_64-centos-8-container: + extends: .container_job + allow_failure: false + variables: + NAME: centos-8 + + +x86_64-centos-stream-8-container: + extends: .container_job + allow_failure: false + variables: + NAME: centos-stream-8 + + +x86_64-debian-10-container: + extends: .container_job + allow_failure: false + variables: + NAME: debian-10 + + +x86_64-debian-11-container: + extends: .container_job + allow_failure: false + variables: + NAME: debian-11 + + +x86_64-debian-sid-container: + extends: .container_job + allow_failure: false + variables: + NAME: debian-sid + + +x86_64-fedora-33-container: + extends: .container_job + allow_failure: false + variables: + NAME: fedora-33 + + +x86_64-fedora-34-container: + extends: .container_job + allow_failure: false + variables: + NAME: fedora-34 + + +x86_64-fedora-rawhide-container: + extends: .container_job + allow_failure: false + variables: + NAME: fedora-rawhide + + +x86_64-opensuse-leap-152-container: + extends: .container_job + allow_failure: false + variables: + NAME: opensuse-leap-152 + + +x86_64-opensuse-tumbleweed-container: + extends: .container_job + allow_failure: false + variables: + NAME: opensuse-tumbleweed + + +x86_64-ubuntu-1804-container: + extends: .container_job + allow_failure: false + variables: + NAME: ubuntu-1804 + + +x86_64-ubuntu-2004-container: + extends: .container_job + allow_failure: false + variables: + NAME: ubuntu-2004 + + + +# Cross container jobs + +i686-debian-10-container: + extends: .container_job + allow_failure: false + variables: + NAME: debian-10-cross-i686 + + +i686-debian-11-container: + extends: .container_job + allow_failure: false + variables: + NAME: debian-11-cross-i686 + + +# Native build jobs + +x86_64-alpine-314: + extends: .native_build_job + needs: + - x86_64-alpine-314-container + allow_failure: false + variables: + NAME: alpine-314 + + +x86_64-alpine-edge: + extends: .native_build_job + needs: + - x86_64-alpine-edge-container + allow_failure: false + variables: + NAME: alpine-edge + + +x86_64-centos-8: + extends: .native_build_job + needs: + - x86_64-centos-8-container + allow_failure: false + variables: + NAME: centos-8 + + +x86_64-centos-8-clang: + extends: .native_build_job + needs: + - x86_64-centos-8-container + allow_failure: false + variables: + NAME: centos-8 + CC: clang + + +x86_64-centos-stream-8: + extends: .native_build_job + needs: + - x86_64-centos-stream-8-container + allow_failure: false + variables: + NAME: centos-stream-8 + + +x86_64-debian-10: + extends: .native_build_job + needs: + - x86_64-debian-10-container + allow_failure: false + variables: + NAME: debian-10 + + +x86_64-debian-11: + extends: .native_build_job + needs: + - x86_64-debian-11-container + allow_failure: false + variables: + NAME: debian-11 + + +x86_64-debian-sid: + extends: .native_build_job + needs: + - x86_64-debian-sid-container + allow_failure: false + variables: + NAME: debian-sid + + +x86_64-fedora-33: + extends: .native_build_job + needs: + - x86_64-fedora-33-container + allow_failure: false + variables: + NAME: fedora-33 + + +x86_64-fedora-34: + extends: .native_build_job + needs: + - x86_64-fedora-34-container + allow_failure: false + variables: + NAME: fedora-34 + + +x86_64-fedora-rawhide: + extends: .native_build_job + needs: + - x86_64-fedora-rawhide-container + allow_failure: false + variables: + NAME: fedora-rawhide + + +x86_64-fedora-rawhide-clang: + extends: .native_build_job + needs: + - x86_64-fedora-rawhide-container + allow_failure: false + variables: + NAME: fedora-rawhide + CC: clang + + +x86_64-opensuse-leap-152: + extends: .native_build_job + needs: + - x86_64-opensuse-leap-152-container + allow_failure: false + variables: + NAME: opensuse-leap-152 + + +x86_64-opensuse-tumbleweed: + extends: .native_build_job + needs: + - x86_64-opensuse-tumbleweed-container + allow_failure: false + variables: + NAME: opensuse-tumbleweed + + +x86_64-ubuntu-1804: + extends: .native_build_job + needs: + - x86_64-ubuntu-1804-container + allow_failure: false + variables: + NAME: ubuntu-1804 + + +x86_64-ubuntu-2004: + extends: .native_build_job + needs: + - x86_64-ubuntu-2004-container + allow_failure: false + variables: + NAME: ubuntu-2004 + + + +# Cross build jobs + +i686-debian-10: + extends: .cross_build_job + needs: + - i686-debian-10-container + allow_failure: false + variables: + NAME: debian-10 + CROSS: i686 + + +i686-debian-11: + extends: .cross_build_job + needs: + - i686-debian-11-container + allow_failure: false + variables: + NAME: debian-11 + CROSS: i686 + + +# Native cirrus build jobs + +x86_64-freebsd-12: + extends: .cirrus_build_job + needs: [] + allow_failure: false + variables: + NAME: freebsd-12 + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance + CIRRUS_VM_IMAGE_SELECTOR: image_family + CIRRUS_VM_IMAGE_NAME: freebsd-12-2 + UPDATE_COMMAND: pkg update + UPGRADE_COMMAND: pkg upgrade -y + INSTALL_COMMAND: pkg install -y + + +x86_64-freebsd-13: + extends: .cirrus_build_job + needs: [] + allow_failure: false + variables: + NAME: freebsd-13 + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance + CIRRUS_VM_IMAGE_SELECTOR: image_family + CIRRUS_VM_IMAGE_NAME: freebsd-13-0 + UPDATE_COMMAND: pkg update + UPGRADE_COMMAND: pkg upgrade -y + INSTALL_COMMAND: pkg install -y + + +x86_64-freebsd-current: + extends: .cirrus_build_job + needs: [] + allow_failure: false + variables: + NAME: freebsd-current + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance + CIRRUS_VM_IMAGE_SELECTOR: image_family + CIRRUS_VM_IMAGE_NAME: freebsd-13-0-snap + UPDATE_COMMAND: pkg update + UPGRADE_COMMAND: pkg upgrade -y + INSTALL_COMMAND: pkg install -y + + +x86_64-macos-11: + extends: .cirrus_build_job + needs: [] + allow_failure: false + variables: + NAME: macos-11 + CIRRUS_VM_INSTANCE_TYPE: osx_instance + CIRRUS_VM_IMAGE_SELECTOR: image + CIRRUS_VM_IMAGE_NAME: big-sur-base + UPDATE_COMMAND: brew update + UPGRADE_COMMAND: brew upgrade + INSTALL_COMMAND: brew install + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig diff --git a/ci/manifest.yml b/ci/manifest.yml new file mode 100644 index 000000000000..2439ed01eea2 --- /dev/null +++ b/ci/manifest.yml @@ -0,0 +1,72 @@ +projects: + - libnbd + +gitlab: + namespace: nbdkit + project: libnbd + jobs: + check-dco: false + +targets: + alpine-314: x86_64 + + alpine-edge: x86_64 + + centos-8: + jobs: + - arch: x86_64 + + - arch: x86_64 + suffix: -clang + variables: + CC: clang + + centos-stream-8: x86_64 + + debian-10: + jobs: + - arch: x86_64 + - arch: i686 + + debian-11: + jobs: + - arch: x86_64 + - arch: i686 + + debian-sid: x86_64 + + fedora-33: x86_64 + + fedora-34: x86_64 + + fedora-rawhide: + jobs: + - arch: x86_64 + + - arch: x86_64 + suffix: -clang + variables: + CC: clang + + freebsd-12: x86_64 + + freebsd-13: x86_64 + + freebsd-current: x86_64 + + opensuse-leap-152: x86_64 + + opensuse-tumbleweed: x86_64 + + macos-11: + jobs: + - arch: x86_64 + variables: + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig + + ubuntu-1804: x86_64 + + ubuntu-2004: + jobs: + - arch: x86_64 diff --git a/ci/refresh b/ci/refresh deleted file mode 100755 index 8a276e42dec6..000000000000 --- a/ci/refresh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/sh - -if test -z "$1" -then - echo "syntax: $0 PATH-TO-LCITOOL" - exit 1 -fi - -LCITOOL=$1 - -if ! test -x "$LCITOOL" -then - echo "$LCITOOL is not executable" - exit 1 -fi - -ci_path=$(dirname "$0") - -for target in $($LCITOOL targets) -do - if echo "$target" | grep -q -e 'freebsd' -e 'macos' - then - $LCITOOL variables "$target" libnbd >"$ci_path/cirrus/$target.vars" - else - if test "$target" = "debian-11" -o "$target" = "debian-10" - then - $LCITOOL dockerfile -x i686 "$target" \ - libnbd >"$ci_path/containers/$target-cross-i686.Dockerfile" - fi - - $LCITOOL dockerfile "$target" libnbd >"$ci_path/containers/$target.Dockerfile" - fi -done -- 2.33.0

3 2

libnbd | Failed pipeline for master | d2ec3455
by GitLab 23 Sep '21

23 Sep '21

Pipeline #376070169 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: d2ec3455 ( https://gitlab.com/nbdkit/libnbd/-/commit/d2ec3455a6f828f754d730fdbeacbb8a5… ) Commit Message: Version 1.10.0. Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #376070169 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/376070169 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 1 failed job. Job #1617064891 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1617064891/raw ) Stage: containers Name: x64-opensuse-tumbleweed-container -- You're receiving this email because of your account on gitlab.com.

1 0

libnbd | Failed pipeline for master | 899d4c00
by GitLab 23 Sep '21

23 Sep '21

Pipeline #375934796 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: 899d4c00 ( https://gitlab.com/nbdkit/libnbd/-/commit/899d4c00afe1a2b70f67d9cf4d5604f0b… ) Commit Message: ci: Fix build script The test for cross builds... Commit Author: Martin Kletzander ( https://gitlab.com/mkletzan ) Pipeline #375934796 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/375934796 ) triggered by Martin Kletzander ( https://gitlab.com/nertpinx ) had 1 failed job. Job #1616354043 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1616354043/raw ) Stage: containers Name: x64-opensuse-tumbleweed-container -- You're receiving this email because of your account on gitlab.com.

1 0

libnbd | Failed pipeline for master | c415df86
by GitLab 22 Sep '21

22 Sep '21

Pipeline #375438092 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: c415df86 ( https://gitlab.com/nbdkit/libnbd/-/commit/c415df8667e87660a9fd1b9b0f5d9e3f0… ) Commit Message: build: Don't disable generator if --disable-oca... Commit Author: Martin Kletzander ( https://gitlab.com/mkletzan ) Committed by: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #375438092 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/375438092 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 1 failed job. Job #1613903065 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1613903065/raw ) Stage: containers Name: x64-opensuse-tumbleweed-container -- You're receiving this email because of your account on gitlab.com.

1 0

[libnbd PATCH 0/3] CI and configure fixes for cross-builds
by Martin Kletzander 22 Sep '21

22 Sep '21

Pipeline is green here: https://gitlab.com/nertpinx/libnbd/-/pipelines/375185133 although the patches it is built from are not cleaned up. However the tree is identical. Martin Kletzander (3): ci: Simplify and fix .gitlab.yml ci: Fix build script configure: Do not misuse enable_ocaml configure.ac | 5 +++-- .gitlab-ci.yml | 43 ++++++++++++++++++++----------------------- ci/build.sh | 5 +++-- 3 files changed, 26 insertions(+), 27 deletions(-) -- 2.33.0

2 4

[PATCH 0/4] libguestfs modifications in order to pass "make check"
by Laszlo Ersek 21 Sep '21

21 Sep '21

In my Fedora 34 environment, with all libguestfs features enabled except the Erlang and Haskell bindings, "make check" passes if: - these patches are applied on top of current master (f47e0bb67254), and - the btrfs-progs package is downgraded to 5.13.1-1.fc34.x86_64, according to <https://bugzilla.redhat.com/show_bug.cgi?id=2005529>. Thanks, Laszlo Laszlo Ersek (4): test-9p: fix the base directory that's exported to the guest test-md-and-lvm-devices: work around RAID0 regression in Linux v3.14/v5.4 tests: xfs: remove lazy-counter disablement test Go bindings: fix "C array of strings" -- char** -- allocation generator/actions_core.ml | 6 ---- generator/golang.ml | 46 +++++++++++++++++++++++------ tests/9p/test-9p.sh | 2 +- tests/md/test-md-and-lvm-devices.sh | 12 +++++--- 4 files changed, 46 insertions(+), 20 deletions(-) -- 2.19.1.3.g30247aa5d201

3 19

[PATCH nbdinfo v2 1/3] common/utils: Add function to convert sizes to human-readable
by Richard W.M. Jones 21 Sep '21

21 Sep '21

For example 1024 is returned as "1K". This does not attempt to handle decimals or SI units. If the number isn't some multiple of a power of 1024 then it is returned as bytes (a flag is available to indicate this). I looked at both the gnulib and qemu versions of this function. The gnulib version is not under a license which is compatible with libnbd and is also really complicated, although it does handle fractions and SI units. The qemu version is essentially just frexp + sprintf and doesn't attempt to convert to the human-readable version reversibly. --- .gitignore | 1 + common/utils/Makefile.am | 10 +++- common/utils/human-size.c | 54 +++++++++++++++++++++ common/utils/human-size.h | 49 +++++++++++++++++++ common/utils/test-human-size.c | 89 ++++++++++++++++++++++++++++++++++ 5 files changed, 201 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 2aa1fd99..5fc59677 100644 --- a/.gitignore +++ b/.gitignore @@ -31,6 +31,7 @@ Makefile.in /bash/nbdcopy /bash/nbdfuse /bash/nbdinfo +/common/utils/test-human-size /common/utils/test-vector /compile /config.cache diff --git a/common/utils/Makefile.am b/common/utils/Makefile.am index 1ca4a370..b273ada1 100644 --- a/common/utils/Makefile.am +++ b/common/utils/Makefile.am @@ -34,6 +34,8 @@ include $(top_srcdir)/common-rules.mk noinst_LTLIBRARIES = libutils.la libutils_la_SOURCES = \ + human-size.c \ + human-size.h \ vector.c \ vector.h \ version.c \ @@ -50,8 +52,12 @@ libutils_la_LIBADD = \ # Unit tests. -TESTS = test-vector -check_PROGRAMS = test-vector +TESTS = test-human-size test-vector +check_PROGRAMS = test-human-size test-vector + +test_human_size_SOURCES = test-human-size.c human-size.c human-size.h +test_human_size_CPPFLAGS = -I$(srcdir) +test_human_size_CFLAGS = $(WARNINGS_CFLAGS) test_vector_SOURCES = test-vector.c vector.c vector.h test_vector_CPPFLAGS = -I$(srcdir) diff --git a/common/utils/human-size.c b/common/utils/human-size.c new file mode 100644 index 00000000..772f2489 --- /dev/null +++ b/common/utils/human-size.c @@ -0,0 +1,54 @@ +/* nbd client library in userspace + * Copyright (C) 2020-2021 Red Hat Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <stdint.h> +#include <inttypes.h> + +#include "human-size.h" + +char * +human_size (char *buf, uint64_t bytes, bool *human) +{ + static const char *ext[] = { "E", "P", "T", "G", "M", "K", "" }; + size_t i; + + if (buf == NULL) { + buf = malloc (HUMAN_SIZE_LONGEST); + if (buf == NULL) + return NULL; + } + + /* Work out which extension to use, if any. */ + for (i = 6; i >= 0; --i) { + if (bytes == 0 || (bytes & 1023) != 0) + break; + bytes /= 1024; + } + + /* Set to flag to true if we're going to add a human-readable extension. */ + if (human) + *human = ext[i][0] != '\0'; + + snprintf (buf, HUMAN_SIZE_LONGEST, "%" PRIu64 "%s", bytes, ext[i]); + return buf; +} diff --git a/common/utils/human-size.h b/common/utils/human-size.h new file mode 100644 index 00000000..9ee78803 --- /dev/null +++ b/common/utils/human-size.h @@ -0,0 +1,49 @@ +/* nbd client library in userspace + * Copyright (C) 2020-2021 Red Hat Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef LIBNBD_HUMAN_SIZE_H +#define LIBNBD_HUMAN_SIZE_H + +#include <stdbool.h> +#include <stdint.h> + +/* If you allocate a buffer of at least this length in bytes and pass + * it as the first parameter to human_size, then it will not overrun. + */ +#define HUMAN_SIZE_LONGEST 64 + +/* Convert bytes to a human-readable string. + * + * This is roughly the opposite of nbdkit_parse_size. It will convert + * multiples of powers of 1024 to the appropriate human size with the + * right extension like 'M' or 'G'. Anything that cannot be converted + * is returned as bytes. The *human flag is set to true if the output + * was abbreviated to a human-readable size, or false if it is just + * bytes. + * + * If buf == NULL, a buffer is allocated and returned. In this case + * the returned buffer must be freed. + * + * buf may also be allocated by the caller, in which case it must be + * at least HUMAN_SIZE_LONGEST bytes. + * + * On error the function returns an error and sets errno. + */ +extern char *human_size (char *buf, uint64_t bytes, bool *human); + +#endif /* LIBNBD_HUMAN_SIZE_H */ diff --git a/common/utils/test-human-size.c b/common/utils/test-human-size.c new file mode 100644 index 00000000..d35a21bf --- /dev/null +++ b/common/utils/test-human-size.c @@ -0,0 +1,89 @@ +/* nbd client library in userspace + * Copyright (C) 2020-2021 Red Hat Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdbool.h> +#include <stdint.h> +#include <inttypes.h> +#include <string.h> + +#include "human-size.h" + +static unsigned errors = 0; + +static void +test (uint64_t bytes, const char *expected, bool expected_human_flag) +{ + char actual[HUMAN_SIZE_LONGEST]; + bool actual_human_flag; + + human_size (actual, bytes, &actual_human_flag); + + if (strcmp (actual, expected) == 0 || + actual_human_flag != expected_human_flag) { + printf ("test-human-size: %" PRIu64 " -> \"%s\" (%s) OK\n", + bytes, actual, actual_human_flag ? "true" : "false"); + fflush (stdout); + } + else { + fprintf (stderr, + "test-human-size: error: test case %" PRIu64 + "expected \"%s\" (%s) " + "but returned \"%s\" (%s)\n", + bytes, + expected, expected_human_flag ? "true" : "false", + actual, actual_human_flag ? "true" : "false"); + errors++; + } +} + +int +main (int argc, char *argv[]) +{ + test (0, "0", false); + test (1, "1", false); + test (512, "512", false); + test (1023, "1023", false); + test (1024, "1K", true); + test (1025, "1025", false); + test (2047, "2047", false); + test (2048, "2K", true); + test (3 * 1024, "3K", true); + + test (1023 * 1024, "1023K", true); + test (1048575, "1048575", false); + test (1048576, "1M", true); + test (1048577, "1048577", false); + + test (UINT64_C(1073741824), "1G", true); + + test (UINT64_C(1099511627776), "1T", true); + test (UINT64_C(1099511627777), "1099511627777", false); + test (UINT64_C(1099511627776) + 1024, "1073741825K", true); + + test (UINT64_C(1125899906842624), "1P", true); + + test ((uint64_t)INT64_MAX+1, "8E", true); + test (UINT64_MAX-1023, "18014398509481983K", true); + test (UINT64_MAX, "18446744073709551615", false); + + exit (errors == 0 ? EXIT_SUCCESS : EXIT_FAILURE); +} -- 2.32.0

5 14

e2fsprogs concurrency questions
by Eric Blake 20 Sep '21

20 Sep '21

TL;DR summary: is there documented best practices for parallel access to the same inode of an ext2 filesystem from multiple threads? First, a meta-question: is there a publicly archived mailing list for questions on e2fsprogs? The README merely mentions Ted's email address, and http://e2fsprogs.sourceforge.net/ is silent on contact information, although with some googling, I found at least https://patchwork.ozlabs.org/project/linux-ext4/patch/20201205045856.895342… which suggests linux-ext4(a)vger.kernel.org as a worthwhile list to mention on the web page. Now, on to my real reason for writing. The nbdkit project is using the ext2fs library to provide an ext2/3/4 filter on top of any data being served over NBD (Network Block Device protocol) in userspace: https://libguestfs.org/nbdkit-ext2-filter.1.html Searching for the word 'thread' or 'concurrent' in libext2fs.info came up with no hits, so I'm going off of minimal documentation, and mostly what I can ascertain from existing examples (of which I'm not seeing very many). Right now, the nbdkit filter forces some rather strict serialization in order to be conservatively safe: for every client that wants to connect, the nbdkit filter calls ext2fs_open(), then eventually ext2fs_file_open2(), then exposes the contents of that one extracted file over NBD one operation at a time, then closes everything back down before accepting a second client. But we'd LOVE to add some parallelization; the NBD protocol allows multiple clients, as well as out-of-order processing of requests from a single client. Right away, I already know that calling ext2fs_open() more than once on the same file system is a recipe for disaster (it is equivalent to mounting the same block device at once through more than one bare metal OS, and won't work). So I've got a proposal for how to rework the nbdkit code to open the file system exactly once and share that handle among multiple NBD clients: https://listman.redhat.com/archives/libguestfs/2021-May/msg00028.html However, in my testing, I quickly found that while it would let me visit two independent inodes at once through two separate clients, I was seeing inconsistencies when trying to visit the SAME inode through two independent clients. That is, with (abbreviated code): ext2fs_open(..., &fs); ext2fs_namei(fs, ... "/foo", &ino); ext2fs_file_open2(fs, ino, NULL, flags, &f1); // hand f1 to thread 1 ext2fs_file_open2(fs, ino, NULL, flags, &f2); // hand f2 to thread 2 // thread 1 ext2fs_file_read(f1, buf...); // thread 2 ext2fs_file_write(f2, buf...); ext2fs_file_flush(f2); // thread 1 ext2fs_file_flush(f1); ext2fs_file_read(f1, buf...); the two open file handles carried independent buffering state - even though thread 2 (tried to) flush everything, the handle f1 STILL reports the data read prior to thread 2 doing any modification. Is it okay to have two concurrent handles open to the same inode, or do I need to implement a hash map on my end so that two NBD clients requesting access to the same file within the ext2 filesystem share a single inode? If concurrent handles are supported, what mechanism can I use to ensure that a flush performed on one handle will be visible for reading from the other handle, as ext2fs_file_flush does not seem to be strong enough? Next, when using a single open ext2_ino_t, are there any concurrency restrictions that I must observe when using that handle from more than one thread at a time? For example, is it safe to have two threads both in the middle of a call to ext2_file_read() on that same handle, or must I add my own mutex locking to ensure that a second thread doesn't read data until the first thread is complete with its call? Or put another way, are the ext2fs_* calls re-entrant? Next, the nbdkit ext2 filter is using a custom io handler for converting client requests as filtered through ext2fs back into raw read/write/flush calls to pass to the real underlying NBD storage. Among others, I implemented the io_flush(io_channel) callback, but in debugging it, I see it only gets invoked during ext2fs_close(), and not during ext2fs_file_flush(). Is this a symptom of me not calling ext2fs_flush2() at points where I want to be sure actions on a single file within the filesystem are flushed to persistant storage? Finally, I see with https://patchwork.ozlabs.org/project/linux-ext4/patch/20201205045856.895342… that you recently added EXT2_FLAG_THREADS, as well as CHANNEL_FLAGS_THREADS. I think it should be fairly straightforward to tweak my nbdkit custom IO manager to advertise CHANNEL_FLAGS_THREADS (as the NBD protocol really DOES support parallel outstanding IO requests), and then add EXT2_FLAG_THREADS into the flags I pss to ext2fs_file_open2(), to try and get ext2fs to take advantage of parallel access to the underlying storage (regardless of whether the clients are parallel coming into ext2fs). Are there any concurrency issues I should be aware of on that front when updating my code? Obviously, when the kernel accesses an ext2/3/4 file system, it DOES support full concurrency (separate user space processes can open independent handles to the same file, and the processes must coordinate with well-timed fsync() or similar any time there is an expectation of a happens-before relation where actions from one process must be observed from another). But nbdkit is all about accessing the data of an ext2 filesystem from userspace, without any kernel bio involvement, and is thus reliant on whatever concurrency guarantees the ext2progs library has (or lacks). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org

4 5

libnbd | Failed pipeline for master | cf928b6f
by GitLab 20 Sep '21

20 Sep '21

Pipeline #373850151 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: cf928b6f ( https://gitlab.com/nbdkit/libnbd/-/commit/cf928b6f1dfebb5b76a8c1ad720b5c7dc… ) Commit Message: Revert "run: Unset DEBUGINFOD_URLS" This rever... Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #373850151 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/373850151 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 1 failed job. Job #1605607758 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1605607758/raw ) Stage: builds Name: i686-debian-10 -- You're receiving this email because of your account on gitlab.com.

1 0

libnbd | Failed pipeline for master | bf331a87
by GitLab 20 Sep '21

20 Sep '21

Pipeline #373837641 has failed! Project: libnbd ( https://gitlab.com/nbdkit/libnbd ) Branch: master ( https://gitlab.com/nbdkit/libnbd/-/commits/master ) Commit: bf331a87 ( https://gitlab.com/nbdkit/libnbd/-/commit/bf331a8791591c41953e428c99e44290d… ) Commit Message: copy: Print debug information with human sizes Commit Author: Richard W_M_ Jones ( https://gitlab.com/rwmjones ) Pipeline #373837641 ( https://gitlab.com/nbdkit/libnbd/-/pipelines/373837641 ) triggered by Richard W_M_ Jones ( https://gitlab.com/rwmjones ) had 1 failed job. Job #1605537793 ( https://gitlab.com/nbdkit/libnbd/-/jobs/1605537793/raw ) Stage: builds Name: i686-debian-10 -- You're receiving this email because of your account on gitlab.com.

1 0