[libnbd PATCH] ci: Adopt libvirt-ci's manifest approach
by Martin Kletzander
This makes it a bit easier to write .gitlab-ci.yml as it inherits most of what
can be generated from the ci/gitlab.yml and deprecates the ci/refresh script.
Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com>
---
.gitlab-ci.yml | 270 +-----------
ci/README.rst | 7 +-
ci/cirrus/freebsd-12.vars | 2 +-
ci/cirrus/freebsd-13.vars | 2 +-
ci/cirrus/freebsd-current.vars | 2 +-
ci/cirrus/macos-11.vars | 2 +-
ci/containers/alpine-314.Dockerfile | 2 +-
ci/containers/alpine-edge.Dockerfile | 2 +-
ci/containers/centos-8.Dockerfile | 2 +-
ci/containers/centos-stream-8.Dockerfile | 2 +-
ci/containers/debian-10-cross-i686.Dockerfile | 2 +-
ci/containers/debian-10.Dockerfile | 2 +-
ci/containers/debian-11-cross-i686.Dockerfile | 2 +-
ci/containers/debian-11.Dockerfile | 2 +-
ci/containers/debian-sid.Dockerfile | 2 +-
ci/containers/fedora-33.Dockerfile | 2 +-
ci/containers/fedora-34-cross-i686.Dockerfile | 0
ci/containers/fedora-34.Dockerfile | 2 +-
ci/containers/fedora-rawhide.Dockerfile | 2 +-
ci/containers/opensuse-leap-152.Dockerfile | 2 +-
ci/containers/opensuse-tumbleweed.Dockerfile | 2 +-
ci/containers/ubuntu-1804.Dockerfile | 2 +-
ci/containers/ubuntu-2004.Dockerfile | 2 +-
ci/gitlab.yml | 413 ++++++++++++++++++
ci/manifest.yml | 72 +++
ci/refresh | 33 --
26 files changed, 520 insertions(+), 315 deletions(-)
delete mode 100644 ci/containers/fedora-34-cross-i686.Dockerfile
create mode 100644 ci/gitlab.yml
create mode 100644 ci/manifest.yml
delete mode 100755 ci/refresh
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8976015b1e20..122bc0860178 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -13,30 +13,10 @@ stages:
export CCACHE_MAXSIZE="500M"
export PATH="$CCACHE_WRAPPERSDIR:$PATH"
-# Common templates
-
-.container_job:
- image: docker:stable
- stage: containers
- needs: []
- services:
- - name: registry.gitlab.com/libvirt/libvirt-ci/docker-dind:master
- alias: docker
- before_script:
- - export TAG="$CI_REGISTRY_IMAGE/$NAME:latest"
- - export COMMON_TAG="$CI_REGISTRY/nbdkit/libnbd/$NAME:latest"
- - docker info
- - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
- script:
- - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
- - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/containers/$NAME.Dockerfile" ci/containers
- - docker push "$TAG"
- after_script:
- - docker logout
+include: '/ci/gitlab.yml'
.base_build_job:
stage: builds
- image: $CI_REGISTRY_IMAGE/$NAME:latest
cache:
paths:
- ccache/
@@ -52,6 +32,16 @@ stages:
public: true
when: always
+.native_build_job:
+ extends:
+ - .base_build_job
+ - .gitlab_native_build_job
+
+.cross_build_job:
+ extends:
+ - .base_build_job
+ - .gitlab_cross_build_job
+
# Jobs that we delegate to Cirrus CI because they require an operating
# system other than Linux. These jobs will only run if the required
# setup has been performed on the GitLab account (see ci/README.rst).
@@ -91,241 +81,3 @@ stages:
- cirrus-run -v --show-build-log always ci/cirrus/$NAME.yml
rules:
- if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
-
-# Cross-compilation container build jobs
-i686-debian-10-container:
- extends: .container_job
- variables:
- NAME: debian-10-cross-i686
-
-i686-debian-11-container:
- extends: .container_job
- variables:
- NAME: debian-11-cross-i686
-
-# Native container build jobs
-x64-alpine-314-container:
- extends: .container_job
- variables:
- NAME: alpine-314
-
-x64-alpine-edge-container:
- extends: .container_job
- variables:
- NAME: alpine-edge
-
-x64-centos-8-container:
- extends: .container_job
- variables:
- NAME: centos-8
-
-x64-centos-stream-8-container:
- extends: .container_job
- variables:
- NAME: centos-stream-8
-
-x64-debian-11-container:
- extends: .container_job
- variables:
- NAME: debian-11
-
-x64-debian-sid-container:
- extends: .container_job
- variables:
- NAME: debian-sid
-
-x64-fedora-33-container:
- extends: .container_job
- variables:
- NAME: fedora-33
-
-x64-fedora-34-container:
- extends: .container_job
- variables:
- NAME: fedora-34
-
-x64-fedora-rawhide-container:
- extends: .container_job
- allow_failure: true
- variables:
- NAME: fedora-rawhide
-
-x64-opensuse-leap-152-container:
- extends: .container_job
- variables:
- NAME: opensuse-leap-152
-
-x64-opensuse-tumbleweed-container:
- extends: .container_job
- variables:
- NAME: opensuse-tumbleweed
-
-x64-ubuntu-1804-container:
- extends: .container_job
- variables:
- NAME: ubuntu-1804
-
-x64-ubuntu-2004-container:
- extends: .container_job
- variables:
- NAME: ubuntu-2004
-
-# Cross-compilation build jobs
-i686-debian-10:
- extends: .base_build_job
- needs:
- - i686-debian-10-container
- variables:
- NAME: debian-10-cross-i686
- CROSS: i686
-
-i686-debian-11:
- extends: .base_build_job
- needs:
- - i686-debian-11-container
- variables:
- NAME: debian-11-cross-i686
- CROSS: i686
-
-# Native architecture build + test jobs
-x64-debian-11:
- extends: .base_build_job
- needs:
- - x64-debian-11-container
- variables:
- NAME: debian-11
-
-x64-debian-sid:
- extends: .base_build_job
- needs:
- - x64-debian-sid-container
- variables:
- NAME: debian-sid
-
-x64-centos-8:
- extends: .base_build_job
- needs:
- - x64-centos-8-container
- variables:
- NAME: centos-8
-
-x64-alpine-314:
- extends: .base_build_job
- needs:
- - x64-alpine-314-container
- variables:
- NAME: alpine-314
-
-x64-alpine-edge:
- extends: .base_build_job
- needs:
- - x64-alpine-edge-container
- variables:
- NAME: alpine-edge
-
-x64-centos-8-clang:
- extends: .base_build_job
- needs:
- - x64-centos-8-container
- variables:
- NAME: centos-8
- CC: clang
-
-x64-centos-stream-8:
- extends: .base_build_job
- needs:
- - x64-centos-stream-8-container
- variables:
- NAME: centos-stream-8
-
-x64-fedora-33:
- extends: .base_build_job
- needs:
- - x64-fedora-33-container
- variables:
- NAME: fedora-33
-
-x64-fedora-34:
- extends: .base_build_job
- needs:
- - x64-fedora-34-container
- variables:
- NAME: fedora-34
-
-x64-fedora-rawhide:
- extends: .base_build_job
- needs:
- - x64-fedora-rawhide-container
- variables:
- NAME: fedora-rawhide
-
-x64-fedora-rawhide-clang:
- extends: .base_build_job
- needs:
- - x64-fedora-rawhide-container
- variables:
- NAME: fedora-rawhide
- CC: clang
-
-x64-opensuse-leap-152:
- extends: .base_build_job
- needs:
- - x64-opensuse-leap-152-container
- variables:
- NAME: opensuse-leap-152
-
-x64-opensuse-tumbleweed:
- extends: .base_build_job
- needs:
- - x64-opensuse-tumbleweed-container
- variables:
- NAME: opensuse-tumbleweed
-
-x64-ubuntu-1804:
- extends: .base_build_job
- needs:
- - x64-ubuntu-1804-container
- variables:
- NAME: ubuntu-1804
- GOLANG: skip
-
-x64-ubuntu-2004:
- extends: .base_build_job
- needs:
- - x64-ubuntu-2004-container
- variables:
- NAME: ubuntu-2004
-
-x64-freebsd-12-build:
- extends: .cirrus_build_job
- variables:
- NAME: freebsd-12
- CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
- CIRRUS_VM_IMAGE_SELECTOR: image_family
- CIRRUS_VM_IMAGE_NAME: freebsd-12-2
- UPDATE_COMMAND: pkg update
- INSTALL_COMMAND: pkg install -y
-
-x64-freebsd-13-build:
- extends: .cirrus_build_job
- variables:
- NAME: freebsd-13
- CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
- CIRRUS_VM_IMAGE_SELECTOR: image_family
- CIRRUS_VM_IMAGE_NAME: freebsd-13-0
- UPDATE_COMMAND: pkg update
- INSTALL_COMMAND: pkg install -y
-
-x64-macos-11-build:
- extends: .cirrus_build_job
- allow_failure: true
- variables:
- NAME: macos-11
- CIRRUS_VM_INSTANCE_TYPE: osx_instance
- CIRRUS_VM_IMAGE_SELECTOR: image
- CIRRUS_VM_IMAGE_NAME: big-sur-base
- UPDATE_COMMAND: brew update
- UPGRADE_COMMAND: brew upgrade
- INSTALL_COMMAND: brew install
- PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin
- PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
diff --git a/ci/README.rst b/ci/README.rst
index c1c97362e41f..3d11e3a4d8bc 100644
--- a/ci/README.rst
+++ b/ci/README.rst
@@ -35,9 +35,10 @@ Container images are prepared during the build and cached, so that they do not
need to be rebuilt for every run, or even built from scratch when changed
slightly. The cached container images can be deleted at any time as they will
be correctly rebuilt in case of a cache miss. The Dockerfiles are generated
-using ``lcitool`` and stored under ``ci/containers``. In that directory is also
-a ``refresh`` script which can regenerate those files, for example after an
-update to libvirt-ci (e.g. when adding a new dependency).
+using ``lcitool`` and stored under ``ci/containers``. In order to update the
+files one needs to run ``lcitool manifest ci/manifest.yml`` from the root of the
+git repository, for example after an update to libvirt-ci (e.g. when adding a
+new dependency).
Recreating the builds locally is pretty straightforward. Choose a file from
``ci/containers`` which represents the desired setup. In this example let's
diff --git a/ci/cirrus/freebsd-12.vars b/ci/cirrus/freebsd-12.vars
index 2573de241849..6f2b76ce5d11 100644
--- a/ci/cirrus/freebsd-12.vars
+++ b/ci/cirrus/freebsd-12.vars
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool variables freebsd-12 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/cirrus/freebsd-13.vars b/ci/cirrus/freebsd-13.vars
index 40e8ddd43e49..6f2b76ce5d11 100644
--- a/ci/cirrus/freebsd-13.vars
+++ b/ci/cirrus/freebsd-13.vars
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool variables freebsd-13 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/cirrus/freebsd-current.vars b/ci/cirrus/freebsd-current.vars
index 73dff989ffb4..6f2b76ce5d11 100644
--- a/ci/cirrus/freebsd-current.vars
+++ b/ci/cirrus/freebsd-current.vars
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool variables freebsd-current libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/cirrus/macos-11.vars b/ci/cirrus/macos-11.vars
index eae75535a960..89bda78fd154 100644
--- a/ci/cirrus/macos-11.vars
+++ b/ci/cirrus/macos-11.vars
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool variables macos-11 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/alpine-314.Dockerfile b/ci/containers/alpine-314.Dockerfile
index 0fb954abb66b..82960f71ecff 100644
--- a/ci/containers/alpine-314.Dockerfile
+++ b/ci/containers/alpine-314.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile alpine-314 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/alpine-edge.Dockerfile b/ci/containers/alpine-edge.Dockerfile
index e27775d647fb..3e9fbfa223eb 100644
--- a/ci/containers/alpine-edge.Dockerfile
+++ b/ci/containers/alpine-edge.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile alpine-edge libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/centos-8.Dockerfile b/ci/containers/centos-8.Dockerfile
index 1e05d4cda98a..d46b428f726c 100644
--- a/ci/containers/centos-8.Dockerfile
+++ b/ci/containers/centos-8.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile centos-8 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/centos-stream-8.Dockerfile b/ci/containers/centos-stream-8.Dockerfile
index 8f2dff57b3a3..613d05d6ec0c 100644
--- a/ci/containers/centos-stream-8.Dockerfile
+++ b/ci/containers/centos-stream-8.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile centos-stream-8 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/debian-10-cross-i686.Dockerfile b/ci/containers/debian-10-cross-i686.Dockerfile
index c7c8dca8df4e..5641f58b71fe 100644
--- a/ci/containers/debian-10-cross-i686.Dockerfile
+++ b/ci/containers/debian-10-cross-i686.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile --cross i686 debian-10 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/debian-10.Dockerfile b/ci/containers/debian-10.Dockerfile
index fbea28350a06..6cc51ec44de2 100644
--- a/ci/containers/debian-10.Dockerfile
+++ b/ci/containers/debian-10.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile debian-10 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/debian-11-cross-i686.Dockerfile b/ci/containers/debian-11-cross-i686.Dockerfile
index e0d4fc15961d..fe19aa21c2ed 100644
--- a/ci/containers/debian-11-cross-i686.Dockerfile
+++ b/ci/containers/debian-11-cross-i686.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile --cross i686 debian-11 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/debian-11.Dockerfile b/ci/containers/debian-11.Dockerfile
index c817870daa73..cb52e7eb4db3 100644
--- a/ci/containers/debian-11.Dockerfile
+++ b/ci/containers/debian-11.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile debian-11 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/debian-sid.Dockerfile b/ci/containers/debian-sid.Dockerfile
index 909b772c473f..e5a3b663b105 100644
--- a/ci/containers/debian-sid.Dockerfile
+++ b/ci/containers/debian-sid.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile debian-sid libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/fedora-33.Dockerfile b/ci/containers/fedora-33.Dockerfile
index 353aa0d1e789..9c0bfa954182 100644
--- a/ci/containers/fedora-33.Dockerfile
+++ b/ci/containers/fedora-33.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile fedora-33 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/fedora-34-cross-i686.Dockerfile b/ci/containers/fedora-34-cross-i686.Dockerfile
deleted file mode 100644
index e69de29bb2d1..000000000000
diff --git a/ci/containers/fedora-34.Dockerfile b/ci/containers/fedora-34.Dockerfile
index 35d84cc3e0f2..b3f57abfd096 100644
--- a/ci/containers/fedora-34.Dockerfile
+++ b/ci/containers/fedora-34.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile fedora-34 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/fedora-rawhide.Dockerfile b/ci/containers/fedora-rawhide.Dockerfile
index acbb61f0e74a..49c3e69cc7d6 100644
--- a/ci/containers/fedora-rawhide.Dockerfile
+++ b/ci/containers/fedora-rawhide.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile fedora-rawhide libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/opensuse-leap-152.Dockerfile b/ci/containers/opensuse-leap-152.Dockerfile
index 050ec442151e..330399253786 100644
--- a/ci/containers/opensuse-leap-152.Dockerfile
+++ b/ci/containers/opensuse-leap-152.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile opensuse-leap-152 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/opensuse-tumbleweed.Dockerfile b/ci/containers/opensuse-tumbleweed.Dockerfile
index bbcdd3220094..96d4298a5db1 100644
--- a/ci/containers/opensuse-tumbleweed.Dockerfile
+++ b/ci/containers/opensuse-tumbleweed.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile opensuse-tumbleweed libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/ubuntu-1804.Dockerfile b/ci/containers/ubuntu-1804.Dockerfile
index 72dc53b39c13..a172a0cbac4e 100644
--- a/ci/containers/ubuntu-1804.Dockerfile
+++ b/ci/containers/ubuntu-1804.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile ubuntu-1804 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/containers/ubuntu-2004.Dockerfile b/ci/containers/ubuntu-2004.Dockerfile
index 6a11fae8d27c..88ea6cfe70dd 100644
--- a/ci/containers/ubuntu-2004.Dockerfile
+++ b/ci/containers/ubuntu-2004.Dockerfile
@@ -1,6 +1,6 @@
# THIS FILE WAS AUTO-GENERATED
#
-# $ lcitool dockerfile ubuntu-2004 libnbd
+# $ lcitool manifest ci/manifest.yml
#
# https://gitlab.com/libvirt/libvirt-ci
diff --git a/ci/gitlab.yml b/ci/gitlab.yml
new file mode 100644
index 000000000000..df3f68ad5587
--- /dev/null
+++ b/ci/gitlab.yml
@@ -0,0 +1,413 @@
+# THIS FILE WAS AUTO-GENERATED
+#
+# $ lcitool manifest ci/manifest.yml
+#
+# https://gitlab.com/libvirt/libvirt-ci
+
+
+.container_job:
+ image: docker:stable
+ stage: containers
+ needs: []
+ services:
+ - name: registry.gitlab.com/libvirt/libvirt-ci/docker-dind:master
+ alias: docker
+ before_script:
+ - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+ - export COMMON_TAG="$CI_REGISTRY/nbdkit/libnbd/ci-$NAME:latest"
+ - docker info
+ - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+ script:
+ - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+ - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/containers/$NAME.Dockerfile" ci/containers
+ - docker push "$TAG"
+ after_script:
+ - docker logout
+
+
+.gitlab_native_build_job:
+ image: $CI_REGISTRY_IMAGE/ci-$NAME:latest
+ stage: builds
+
+
+.gitlab_cross_build_job:
+ image: $CI_REGISTRY_IMAGE/ci-$NAME-cross-$CROSS:latest
+ stage: builds
+
+
+.cirrus_build_job:
+ stage: builds
+ image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
+ needs: []
+ script:
+ - source ci/cirrus/$NAME.vars
+ - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
+ -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
+ -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
+ -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
+ -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
+ -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
+ -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
+ -e "s|[@]UPGRADE_COMMAND@|$UPGRADE_COMMAND|g"
+ -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
+ -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
+ -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
+ -e "s|[@]PKGS@|$PKGS|g"
+ -e "s|[@]MAKE@|$MAKE|g"
+ -e "s|[@]PYTHON@|$PYTHON|g"
+ -e "s|[@]PIP3@|$PIP3|g"
+ -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
+ -e "s|[@]XML_CATALOG_FILES@|$XML_CATALOG_FILES|g"
+ <ci/cirrus/build.yml >ci/cirrus/$NAME.yml
+ - cat ci/cirrus/$NAME.yml
+ - cirrus-run -v --show-build-log always ci/cirrus/$NAME.yml
+ rules:
+ - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
+
+
+# Native container jobs
+
+x86_64-alpine-314-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: alpine-314
+
+
+x86_64-alpine-edge-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: alpine-edge
+
+
+x86_64-centos-8-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: centos-8
+
+
+x86_64-centos-stream-8-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: centos-stream-8
+
+
+x86_64-debian-10-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: debian-10
+
+
+x86_64-debian-11-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: debian-11
+
+
+x86_64-debian-sid-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: debian-sid
+
+
+x86_64-fedora-33-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: fedora-33
+
+
+x86_64-fedora-34-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: fedora-34
+
+
+x86_64-fedora-rawhide-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: fedora-rawhide
+
+
+x86_64-opensuse-leap-152-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: opensuse-leap-152
+
+
+x86_64-opensuse-tumbleweed-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: opensuse-tumbleweed
+
+
+x86_64-ubuntu-1804-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: ubuntu-1804
+
+
+x86_64-ubuntu-2004-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: ubuntu-2004
+
+
+
+# Cross container jobs
+
+i686-debian-10-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: debian-10-cross-i686
+
+
+i686-debian-11-container:
+ extends: .container_job
+ allow_failure: false
+ variables:
+ NAME: debian-11-cross-i686
+
+
+# Native build jobs
+
+x86_64-alpine-314:
+ extends: .native_build_job
+ needs:
+ - x86_64-alpine-314-container
+ allow_failure: false
+ variables:
+ NAME: alpine-314
+
+
+x86_64-alpine-edge:
+ extends: .native_build_job
+ needs:
+ - x86_64-alpine-edge-container
+ allow_failure: false
+ variables:
+ NAME: alpine-edge
+
+
+x86_64-centos-8:
+ extends: .native_build_job
+ needs:
+ - x86_64-centos-8-container
+ allow_failure: false
+ variables:
+ NAME: centos-8
+
+
+x86_64-centos-8-clang:
+ extends: .native_build_job
+ needs:
+ - x86_64-centos-8-container
+ allow_failure: false
+ variables:
+ NAME: centos-8
+ CC: clang
+
+
+x86_64-centos-stream-8:
+ extends: .native_build_job
+ needs:
+ - x86_64-centos-stream-8-container
+ allow_failure: false
+ variables:
+ NAME: centos-stream-8
+
+
+x86_64-debian-10:
+ extends: .native_build_job
+ needs:
+ - x86_64-debian-10-container
+ allow_failure: false
+ variables:
+ NAME: debian-10
+
+
+x86_64-debian-11:
+ extends: .native_build_job
+ needs:
+ - x86_64-debian-11-container
+ allow_failure: false
+ variables:
+ NAME: debian-11
+
+
+x86_64-debian-sid:
+ extends: .native_build_job
+ needs:
+ - x86_64-debian-sid-container
+ allow_failure: false
+ variables:
+ NAME: debian-sid
+
+
+x86_64-fedora-33:
+ extends: .native_build_job
+ needs:
+ - x86_64-fedora-33-container
+ allow_failure: false
+ variables:
+ NAME: fedora-33
+
+
+x86_64-fedora-34:
+ extends: .native_build_job
+ needs:
+ - x86_64-fedora-34-container
+ allow_failure: false
+ variables:
+ NAME: fedora-34
+
+
+x86_64-fedora-rawhide:
+ extends: .native_build_job
+ needs:
+ - x86_64-fedora-rawhide-container
+ allow_failure: false
+ variables:
+ NAME: fedora-rawhide
+
+
+x86_64-fedora-rawhide-clang:
+ extends: .native_build_job
+ needs:
+ - x86_64-fedora-rawhide-container
+ allow_failure: false
+ variables:
+ NAME: fedora-rawhide
+ CC: clang
+
+
+x86_64-opensuse-leap-152:
+ extends: .native_build_job
+ needs:
+ - x86_64-opensuse-leap-152-container
+ allow_failure: false
+ variables:
+ NAME: opensuse-leap-152
+
+
+x86_64-opensuse-tumbleweed:
+ extends: .native_build_job
+ needs:
+ - x86_64-opensuse-tumbleweed-container
+ allow_failure: false
+ variables:
+ NAME: opensuse-tumbleweed
+
+
+x86_64-ubuntu-1804:
+ extends: .native_build_job
+ needs:
+ - x86_64-ubuntu-1804-container
+ allow_failure: false
+ variables:
+ NAME: ubuntu-1804
+
+
+x86_64-ubuntu-2004:
+ extends: .native_build_job
+ needs:
+ - x86_64-ubuntu-2004-container
+ allow_failure: false
+ variables:
+ NAME: ubuntu-2004
+
+
+
+# Cross build jobs
+
+i686-debian-10:
+ extends: .cross_build_job
+ needs:
+ - i686-debian-10-container
+ allow_failure: false
+ variables:
+ NAME: debian-10
+ CROSS: i686
+
+
+i686-debian-11:
+ extends: .cross_build_job
+ needs:
+ - i686-debian-11-container
+ allow_failure: false
+ variables:
+ NAME: debian-11
+ CROSS: i686
+
+
+# Native cirrus build jobs
+
+x86_64-freebsd-12:
+ extends: .cirrus_build_job
+ needs: []
+ allow_failure: false
+ variables:
+ NAME: freebsd-12
+ CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image_family
+ CIRRUS_VM_IMAGE_NAME: freebsd-12-2
+ UPDATE_COMMAND: pkg update
+ UPGRADE_COMMAND: pkg upgrade -y
+ INSTALL_COMMAND: pkg install -y
+
+
+x86_64-freebsd-13:
+ extends: .cirrus_build_job
+ needs: []
+ allow_failure: false
+ variables:
+ NAME: freebsd-13
+ CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image_family
+ CIRRUS_VM_IMAGE_NAME: freebsd-13-0
+ UPDATE_COMMAND: pkg update
+ UPGRADE_COMMAND: pkg upgrade -y
+ INSTALL_COMMAND: pkg install -y
+
+
+x86_64-freebsd-current:
+ extends: .cirrus_build_job
+ needs: []
+ allow_failure: false
+ variables:
+ NAME: freebsd-current
+ CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image_family
+ CIRRUS_VM_IMAGE_NAME: freebsd-13-0-snap
+ UPDATE_COMMAND: pkg update
+ UPGRADE_COMMAND: pkg upgrade -y
+ INSTALL_COMMAND: pkg install -y
+
+
+x86_64-macos-11:
+ extends: .cirrus_build_job
+ needs: []
+ allow_failure: false
+ variables:
+ NAME: macos-11
+ CIRRUS_VM_INSTANCE_TYPE: osx_instance
+ CIRRUS_VM_IMAGE_SELECTOR: image
+ CIRRUS_VM_IMAGE_NAME: big-sur-base
+ UPDATE_COMMAND: brew update
+ UPGRADE_COMMAND: brew upgrade
+ INSTALL_COMMAND: brew install
+ PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin
+ PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
diff --git a/ci/manifest.yml b/ci/manifest.yml
new file mode 100644
index 000000000000..2439ed01eea2
--- /dev/null
+++ b/ci/manifest.yml
@@ -0,0 +1,72 @@
+projects:
+ - libnbd
+
+gitlab:
+ namespace: nbdkit
+ project: libnbd
+ jobs:
+ check-dco: false
+
+targets:
+ alpine-314: x86_64
+
+ alpine-edge: x86_64
+
+ centos-8:
+ jobs:
+ - arch: x86_64
+
+ - arch: x86_64
+ suffix: -clang
+ variables:
+ CC: clang
+
+ centos-stream-8: x86_64
+
+ debian-10:
+ jobs:
+ - arch: x86_64
+ - arch: i686
+
+ debian-11:
+ jobs:
+ - arch: x86_64
+ - arch: i686
+
+ debian-sid: x86_64
+
+ fedora-33: x86_64
+
+ fedora-34: x86_64
+
+ fedora-rawhide:
+ jobs:
+ - arch: x86_64
+
+ - arch: x86_64
+ suffix: -clang
+ variables:
+ CC: clang
+
+ freebsd-12: x86_64
+
+ freebsd-13: x86_64
+
+ freebsd-current: x86_64
+
+ opensuse-leap-152: x86_64
+
+ opensuse-tumbleweed: x86_64
+
+ macos-11:
+ jobs:
+ - arch: x86_64
+ variables:
+ PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin
+ PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
+
+ ubuntu-1804: x86_64
+
+ ubuntu-2004:
+ jobs:
+ - arch: x86_64
diff --git a/ci/refresh b/ci/refresh
deleted file mode 100755
index 8a276e42dec6..000000000000
--- a/ci/refresh
+++ /dev/null
@@ -1,33 +0,0 @@
-#!/bin/sh
-
-if test -z "$1"
-then
- echo "syntax: $0 PATH-TO-LCITOOL"
- exit 1
-fi
-
-LCITOOL=$1
-
-if ! test -x "$LCITOOL"
-then
- echo "$LCITOOL is not executable"
- exit 1
-fi
-
-ci_path=$(dirname "$0")
-
-for target in $($LCITOOL targets)
-do
- if echo "$target" | grep -q -e 'freebsd' -e 'macos'
- then
- $LCITOOL variables "$target" libnbd >"$ci_path/cirrus/$target.vars"
- else
- if test "$target" = "debian-11" -o "$target" = "debian-10"
- then
- $LCITOOL dockerfile -x i686 "$target" \
- libnbd >"$ci_path/containers/$target-cross-i686.Dockerfile"
- fi
-
- $LCITOOL dockerfile "$target" libnbd >"$ci_path/containers/$target.Dockerfile"
- fi
-done
--
2.33.0
2 years, 11 months
[PATCH 0/4] libguestfs modifications in order to pass "make check"
by Laszlo Ersek
In my Fedora 34 environment, with all libguestfs features enabled except
the Erlang and Haskell bindings, "make check" passes if:
- these patches are applied on top of current master (f47e0bb67254), and
- the btrfs-progs package is downgraded to 5.13.1-1.fc34.x86_64,
according to <https://bugzilla.redhat.com/show_bug.cgi?id=2005529>.
Thanks,
Laszlo
Laszlo Ersek (4):
test-9p: fix the base directory that's exported to the guest
test-md-and-lvm-devices: work around RAID0 regression in Linux
v3.14/v5.4
tests: xfs: remove lazy-counter disablement test
Go bindings: fix "C array of strings" -- char** -- allocation
generator/actions_core.ml | 6 ----
generator/golang.ml | 46 +++++++++++++++++++++++------
tests/9p/test-9p.sh | 2 +-
tests/md/test-md-and-lvm-devices.sh | 12 +++++---
4 files changed, 46 insertions(+), 20 deletions(-)
--
2.19.1.3.g30247aa5d201
2 years, 11 months
[PATCH nbdinfo v2 1/3] common/utils: Add function to convert sizes to human-readable
by Richard W.M. Jones
For example 1024 is returned as "1K".
This does not attempt to handle decimals or SI units. If the number
isn't some multiple of a power of 1024 then it is returned as bytes (a
flag is available to indicate this).
I looked at both the gnulib and qemu versions of this function. The
gnulib version is not under a license which is compatible with libnbd
and is also really complicated, although it does handle fractions and
SI units. The qemu version is essentially just frexp + sprintf and
doesn't attempt to convert to the human-readable version reversibly.
---
.gitignore | 1 +
common/utils/Makefile.am | 10 +++-
common/utils/human-size.c | 54 +++++++++++++++++++++
common/utils/human-size.h | 49 +++++++++++++++++++
common/utils/test-human-size.c | 89 ++++++++++++++++++++++++++++++++++
5 files changed, 201 insertions(+), 2 deletions(-)
diff --git a/.gitignore b/.gitignore
index 2aa1fd99..5fc59677 100644
--- a/.gitignore
+++ b/.gitignore
@@ -31,6 +31,7 @@ Makefile.in
/bash/nbdcopy
/bash/nbdfuse
/bash/nbdinfo
+/common/utils/test-human-size
/common/utils/test-vector
/compile
/config.cache
diff --git a/common/utils/Makefile.am b/common/utils/Makefile.am
index 1ca4a370..b273ada1 100644
--- a/common/utils/Makefile.am
+++ b/common/utils/Makefile.am
@@ -34,6 +34,8 @@ include $(top_srcdir)/common-rules.mk
noinst_LTLIBRARIES = libutils.la
libutils_la_SOURCES = \
+ human-size.c \
+ human-size.h \
vector.c \
vector.h \
version.c \
@@ -50,8 +52,12 @@ libutils_la_LIBADD = \
# Unit tests.
-TESTS = test-vector
-check_PROGRAMS = test-vector
+TESTS = test-human-size test-vector
+check_PROGRAMS = test-human-size test-vector
+
+test_human_size_SOURCES = test-human-size.c human-size.c human-size.h
+test_human_size_CPPFLAGS = -I$(srcdir)
+test_human_size_CFLAGS = $(WARNINGS_CFLAGS)
test_vector_SOURCES = test-vector.c vector.c vector.h
test_vector_CPPFLAGS = -I$(srcdir)
diff --git a/common/utils/human-size.c b/common/utils/human-size.c
new file mode 100644
index 00000000..772f2489
--- /dev/null
+++ b/common/utils/human-size.c
@@ -0,0 +1,54 @@
+/* nbd client library in userspace
+ * Copyright (C) 2020-2021 Red Hat Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <inttypes.h>
+
+#include "human-size.h"
+
+char *
+human_size (char *buf, uint64_t bytes, bool *human)
+{
+ static const char *ext[] = { "E", "P", "T", "G", "M", "K", "" };
+ size_t i;
+
+ if (buf == NULL) {
+ buf = malloc (HUMAN_SIZE_LONGEST);
+ if (buf == NULL)
+ return NULL;
+ }
+
+ /* Work out which extension to use, if any. */
+ for (i = 6; i >= 0; --i) {
+ if (bytes == 0 || (bytes & 1023) != 0)
+ break;
+ bytes /= 1024;
+ }
+
+ /* Set to flag to true if we're going to add a human-readable extension. */
+ if (human)
+ *human = ext[i][0] != '\0';
+
+ snprintf (buf, HUMAN_SIZE_LONGEST, "%" PRIu64 "%s", bytes, ext[i]);
+ return buf;
+}
diff --git a/common/utils/human-size.h b/common/utils/human-size.h
new file mode 100644
index 00000000..9ee78803
--- /dev/null
+++ b/common/utils/human-size.h
@@ -0,0 +1,49 @@
+/* nbd client library in userspace
+ * Copyright (C) 2020-2021 Red Hat Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef LIBNBD_HUMAN_SIZE_H
+#define LIBNBD_HUMAN_SIZE_H
+
+#include <stdbool.h>
+#include <stdint.h>
+
+/* If you allocate a buffer of at least this length in bytes and pass
+ * it as the first parameter to human_size, then it will not overrun.
+ */
+#define HUMAN_SIZE_LONGEST 64
+
+/* Convert bytes to a human-readable string.
+ *
+ * This is roughly the opposite of nbdkit_parse_size. It will convert
+ * multiples of powers of 1024 to the appropriate human size with the
+ * right extension like 'M' or 'G'. Anything that cannot be converted
+ * is returned as bytes. The *human flag is set to true if the output
+ * was abbreviated to a human-readable size, or false if it is just
+ * bytes.
+ *
+ * If buf == NULL, a buffer is allocated and returned. In this case
+ * the returned buffer must be freed.
+ *
+ * buf may also be allocated by the caller, in which case it must be
+ * at least HUMAN_SIZE_LONGEST bytes.
+ *
+ * On error the function returns an error and sets errno.
+ */
+extern char *human_size (char *buf, uint64_t bytes, bool *human);
+
+#endif /* LIBNBD_HUMAN_SIZE_H */
diff --git a/common/utils/test-human-size.c b/common/utils/test-human-size.c
new file mode 100644
index 00000000..d35a21bf
--- /dev/null
+++ b/common/utils/test-human-size.c
@@ -0,0 +1,89 @@
+/* nbd client library in userspace
+ * Copyright (C) 2020-2021 Red Hat Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <string.h>
+
+#include "human-size.h"
+
+static unsigned errors = 0;
+
+static void
+test (uint64_t bytes, const char *expected, bool expected_human_flag)
+{
+ char actual[HUMAN_SIZE_LONGEST];
+ bool actual_human_flag;
+
+ human_size (actual, bytes, &actual_human_flag);
+
+ if (strcmp (actual, expected) == 0 ||
+ actual_human_flag != expected_human_flag) {
+ printf ("test-human-size: %" PRIu64 " -> \"%s\" (%s) OK\n",
+ bytes, actual, actual_human_flag ? "true" : "false");
+ fflush (stdout);
+ }
+ else {
+ fprintf (stderr,
+ "test-human-size: error: test case %" PRIu64
+ "expected \"%s\" (%s) "
+ "but returned \"%s\" (%s)\n",
+ bytes,
+ expected, expected_human_flag ? "true" : "false",
+ actual, actual_human_flag ? "true" : "false");
+ errors++;
+ }
+}
+
+int
+main (int argc, char *argv[])
+{
+ test (0, "0", false);
+ test (1, "1", false);
+ test (512, "512", false);
+ test (1023, "1023", false);
+ test (1024, "1K", true);
+ test (1025, "1025", false);
+ test (2047, "2047", false);
+ test (2048, "2K", true);
+ test (3 * 1024, "3K", true);
+
+ test (1023 * 1024, "1023K", true);
+ test (1048575, "1048575", false);
+ test (1048576, "1M", true);
+ test (1048577, "1048577", false);
+
+ test (UINT64_C(1073741824), "1G", true);
+
+ test (UINT64_C(1099511627776), "1T", true);
+ test (UINT64_C(1099511627777), "1099511627777", false);
+ test (UINT64_C(1099511627776) + 1024, "1073741825K", true);
+
+ test (UINT64_C(1125899906842624), "1P", true);
+
+ test ((uint64_t)INT64_MAX+1, "8E", true);
+ test (UINT64_MAX-1023, "18014398509481983K", true);
+ test (UINT64_MAX, "18446744073709551615", false);
+
+ exit (errors == 0 ? EXIT_SUCCESS : EXIT_FAILURE);
+}
--
2.32.0
2 years, 11 months
e2fsprogs concurrency questions
by Eric Blake
TL;DR summary: is there documented best practices for parallel access
to the same inode of an ext2 filesystem from multiple threads?
First, a meta-question: is there a publicly archived mailing list for
questions on e2fsprogs? The README merely mentions Ted's email
address, and http://e2fsprogs.sourceforge.net/ is silent on contact
information, although with some googling, I found at least
https://patchwork.ozlabs.org/project/linux-ext4/patch/20201205045856.8953...
which suggests linux-ext4(a)vger.kernel.org as a worthwhile list to
mention on the web page.
Now, on to my real reason for writing. The nbdkit project is using
the ext2fs library to provide an ext2/3/4 filter on top of any data
being served over NBD (Network Block Device protocol) in userspace:
https://libguestfs.org/nbdkit-ext2-filter.1.html
Searching for the word 'thread' or 'concurrent' in libext2fs.info came
up with no hits, so I'm going off of minimal documentation, and mostly
what I can ascertain from existing examples (of which I'm not seeing
very many).
Right now, the nbdkit filter forces some rather strict serialization
in order to be conservatively safe: for every client that wants to
connect, the nbdkit filter calls ext2fs_open(), then eventually
ext2fs_file_open2(), then exposes the contents of that one extracted
file over NBD one operation at a time, then closes everything back
down before accepting a second client. But we'd LOVE to add some
parallelization; the NBD protocol allows multiple clients, as well as
out-of-order processing of requests from a single client.
Right away, I already know that calling ext2fs_open() more than once
on the same file system is a recipe for disaster (it is equivalent to
mounting the same block device at once through more than one bare
metal OS, and won't work). So I've got a proposal for how to rework
the nbdkit code to open the file system exactly once and share that
handle among multiple NBD clients:
https://listman.redhat.com/archives/libguestfs/2021-May/msg00028.html
However, in my testing, I quickly found that while it would let me
visit two independent inodes at once through two separate clients, I
was seeing inconsistencies when trying to visit the SAME inode through
two independent clients. That is, with (abbreviated code):
ext2fs_open(..., &fs);
ext2fs_namei(fs, ... "/foo", &ino);
ext2fs_file_open2(fs, ino, NULL, flags, &f1); // hand f1 to thread 1
ext2fs_file_open2(fs, ino, NULL, flags, &f2); // hand f2 to thread 2
// thread 1
ext2fs_file_read(f1, buf...);
// thread 2
ext2fs_file_write(f2, buf...);
ext2fs_file_flush(f2);
// thread 1
ext2fs_file_flush(f1);
ext2fs_file_read(f1, buf...);
the two open file handles carried independent buffering state - even
though thread 2 (tried to) flush everything, the handle f1 STILL
reports the data read prior to thread 2 doing any modification.
Is it okay to have two concurrent handles open to the same inode, or
do I need to implement a hash map on my end so that two NBD clients
requesting access to the same file within the ext2 filesystem share a
single inode? If concurrent handles are supported, what mechanism can
I use to ensure that a flush performed on one handle will be visible
for reading from the other handle, as ext2fs_file_flush does not seem
to be strong enough?
Next, when using a single open ext2_ino_t, are there any concurrency
restrictions that I must observe when using that handle from more than
one thread at a time? For example, is it safe to have two threads
both in the middle of a call to ext2_file_read() on that same handle,
or must I add my own mutex locking to ensure that a second thread
doesn't read data until the first thread is complete with its call?
Or put another way, are the ext2fs_* calls re-entrant?
Next, the nbdkit ext2 filter is using a custom io handler for
converting client requests as filtered through ext2fs back into raw
read/write/flush calls to pass to the real underlying NBD storage.
Among others, I implemented the io_flush(io_channel) callback, but in
debugging it, I see it only gets invoked during ext2fs_close(), and
not during ext2fs_file_flush(). Is this a symptom of me not calling
ext2fs_flush2() at points where I want to be sure actions on a single
file within the filesystem are flushed to persistant storage?
Finally, I see with
https://patchwork.ozlabs.org/project/linux-ext4/patch/20201205045856.8953...
that you recently added EXT2_FLAG_THREADS, as well as
CHANNEL_FLAGS_THREADS. I think it should be fairly straightforward to
tweak my nbdkit custom IO manager to advertise CHANNEL_FLAGS_THREADS
(as the NBD protocol really DOES support parallel outstanding IO
requests), and then add EXT2_FLAG_THREADS into the flags I pss to
ext2fs_file_open2(), to try and get ext2fs to take advantage of
parallel access to the underlying storage (regardless of whether the
clients are parallel coming into ext2fs). Are there any concurrency
issues I should be aware of on that front when updating my code?
Obviously, when the kernel accesses an ext2/3/4 file system, it DOES
support full concurrency (separate user space processes can open
independent handles to the same file, and the processes must
coordinate with well-timed fsync() or similar any time there is an
expectation of a happens-before relation where actions from one
process must be observed from another). But nbdkit is all about
accessing the data of an ext2 filesystem from userspace, without any
kernel bio involvement, and is thus reliant on whatever concurrency
guarantees the ext2progs library has (or lacks).
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
2 years, 11 months