May 2021 - Libguestfs - libguestfs List Archives

[PATCH] point users to Libera Chat rather than FreeNode
by Daniel P. Berrangé 29 Jul '21

29 Jul '21

Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- docs/guestfs-faq.pod | 2 +- website/index.html.in | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/guestfs-faq.pod b/docs/guestfs-faq.pod index bea609591..15b1247b0 100644 --- a/docs/guestfs-faq.pod +++ b/docs/guestfs-faq.pod @@ -108,7 +108,7 @@ There is a mailing list, mainly for development, but users are also welcome to ask questions about libguestfs and the virt tools: L<https://www.redhat.com/mailman/listinfo/libguestfs> -You can also talk to us on IRC channel C<#libguestfs> on FreeNode. +You can also talk to us on IRC channel C<#guestfs> on Libera Chat. We're not always around, so please stay in the channel after asking your question and someone will get back to you. diff --git a/website/index.html.in b/website/index.html.in index f469c5eeb..7453129d6 100644 --- a/website/index.html.in +++ b/website/index.html.in @@ -55,8 +55,8 @@ guestfish --ro -i -a disk.img <p> Join us on the <a href="http://www.redhat.com/mailman/listinfo/libguestfs">libguestfs -mailing list</a>, or on IRC channel <code>#libguestfs</code> -on <a href="http://freenode.net/">FreeNode</a>. +mailing list</a>, or on IRC channel <code>#guestfs</code> +on <a href="https://libera.chat/">Libera Chat</a>. </p> </div> -- 2.31.1

3 4

[PATCH v4] fuse: Allow fallocate(FALLOC_FL_ZERO_RANGE)
by Richard W.M. Jones 18 Jun '21

18 Jun '21

v3 -> v4: - Cleans up the commit message. - As the patch itself is unchanged from v3, I did not retest it.

2 4

Re: [Libguestfs] oVirt import from VMware
by Richard W.M. Jones 16 Jun '21

16 Jun '21

[Adding upstream mailing list] On Mon, May 17, 2021 at 12:32:12PM -0400, Alan Daniels wrote: > > Thanks Martin. The logs (/var/log/vdsm/import on the host) > show: > > > > qemu-img: /var/tmp/v2vovle5b34c.qcow2: CURL: Error opening > file: > > Server does not support 'range' (byte ranges). > > This is unexpected, but I'd need to see the full log to be sure. > > Given the little information available (but it's vCenter 7 so it would > fit), it could be: > > https://bugzilla.redhat.com/show_bug.cgi?id=1846238 > https://bugzilla.redhat.com/show_bug.cgi?id=1841038 > > (They're the same bug, we didn't fix it in RHEL 7) > Hey Rich, > > Thanks for the help! Some more info about the environment: > > Oracle Linux 7.9 > # cat /etc/redhat-release > Red Hat Enterprise Linux Server release 7.9 (Maipo) > > OLVM / oVirt 4.3 > > virt-v2v.x86_64 1:1.40.2-10.0.1.el7 > libguestfs.x86_64 1:1.40.2-10.0.1.el7 > qemu-kvm.x86_64 15:4.2.1-6.el7 > nbdkit.x86_64 1.8.0-4.el7 > qemu-block-curl.x86_64 15:4.2.1-6.el7 > > Running on Oracle Cloud. > > VMware is on another cloud provider and they're connected via VPN. > > The logs looks essentially the same as the one from https://bugzilla.redhat.com > /show_bug.cgi?id=1846238 > > I can't really upgrade from 7.9 to 8.3, so is this expected behavior? Are there > some potential workarounds? It's a bug in qemu which you will need to patch in order to do the import using this method. Other methods don't suffer from the bug because they don't use qemu in the same way. I would recommend exploring the “-i ova” method. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

2 8

[PATCH v2v v3 REBASE] rhv-upload: Validate UUIDs passed to -oo rhv-disk-uuid (RHBZ#1789279)
by Martin Kletzander 03 Jun '21

03 Jun '21

The validation helps us fail early and with a sensible error message. The NIL UUID is not valid for oVirt, but other than that there is no other logic in there merely because the UUID types are a matter of the generator and they are just forwarded in this partucular case. On top of that also check that the UUID is not already taken. This makes the code fail with a sensible error message instead of cryptic error from ovirtsdk. Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com> --- v4: - https://listman.redhat.com/archives/libguestfs/2021-May/msg00095.html - Merge the two patches together - Things mentioned in the review are already in, so there is no other change v3: - https://listman.redhat.com/archives/libguestfs/2020-March/msg00084.html - Do the check in precheck - Fix for Lazy evaluation of regexp UUID v2: - https://www.redhat.com/archives/libguestfs/2020-January/msg00221.html - Use EEXIST instead of EINVAL - Put the colliding UUID into the error - Do not evaluate the PCRE needlessly multiple times v1: - https://www.redhat.com/archives/libguestfs/2020-January/msg00184.html v2v/output_rhv_upload.ml | 18 ++++++++++++++++++ v2v/rhv-upload-precheck.py | 10 ++++++++++ 2 files changed, 28 insertions(+) diff --git a/v2v/output_rhv_upload.ml b/v2v/output_rhv_upload.ml index dbef7011b04b..15ba1078019e 100644 --- a/v2v/output_rhv_upload.ml +++ b/v2v/output_rhv_upload.ml @@ -49,6 +49,16 @@ after their uploads (if you do, you must supply one for each disk): -oo rhv-disk-uuid=UUID Disk UUID ") +let is_nonnil_uuid uuid = + let nil_uuid = "00000000-0000-0000-0000-000000000000" in + let rex_uuid = lazy ( + let hex = "[a-fA-F0-9]" in + let str = sprintf "^%s{8}-%s{4}-%s{4}-%s{4}-%s{12}$" hex hex hex hex hex in + PCRE.compile str + ) in + if uuid = nil_uuid then false + else PCRE.matches (Lazy.force rex_uuid) uuid + let parse_output_options options = let rhv_cafile = ref None in let rhv_cluster = ref None in @@ -71,6 +81,8 @@ let parse_output_options options = | "rhv-verifypeer", "" -> rhv_verifypeer := true | "rhv-verifypeer", v -> rhv_verifypeer := bool_of_string v | "rhv-disk-uuid", v -> + if not (is_nonnil_uuid v) then + error (f_"-o rhv-upload: invalid UUID for -oo rhv-disk-uuid"); rhv_disk_uuids := Some (v :: (Option.default [] !rhv_disk_uuids)) | k, _ -> error (f_"-o rhv-upload: unknown output option ‘-oo %s’") k @@ -256,6 +268,12 @@ object error_unless_output_alloc_sparse output_alloc; (* Python code prechecks. *) + let json_params = match rhv_options.rhv_disk_uuids with + | None -> json_params + | Some uuids -> + let ids = List.map (fun uuid -> JSON.String uuid) uuids in + ("rhv_disk_uuids", JSON.List ids) :: json_params + in let precheck_fn = tmpdir // "v2vprecheck.json" in let fd = Unix.openfile precheck_fn [O_WRONLY; O_CREAT] 0o600 in if Python_script.run_command ~stdout_fd:fd diff --git a/v2v/rhv-upload-precheck.py b/v2v/rhv-upload-precheck.py index 2180ea1043ab..c3f74df35f65 100644 --- a/v2v/rhv-upload-precheck.py +++ b/v2v/rhv-upload-precheck.py @@ -97,6 +97,16 @@ if cpu.architecture == types.Architecture.UNDEFINED: raise RuntimeError("The cluster ‘%s’ has an unknown architecture" % (params['rhv_cluster'])) +# Find if any disk already exists with specified UUID. +disks_service = system_service.disks_service() + +for uuid in params.get('rhv_disk_uuids', []): + try: + disk_service = disks_service.disk_service(uuid).get() + raise RuntimeError("Disk with the UUID '%s' already exists" % uuid) + except sdk.NotFoundError: + pass + # Otherwise everything is OK, print a JSON with the results. results = { "rhv_storagedomain_uuid": storage_domain.id, -- 2.31.1

3 3

[PATCH] build: Don't use non-POSIX tests
by Martin Kletzander 01 Jun '21

01 Jun '21

The `test` builtin/binary usually accepts `==` for string comparison, it is mostly accepted for typos and people being used to double equals, but is not documented and not always accepted either. Since autoconf uses the default shell, it might just fail in some cases with: ./configure: 29986: test: xrustc: unexpected operator ./configure: 29990: test: xcargo: unexpected operator Just change it to single equals as it is done everywhere else. Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com> --- m4/guestfs-rust.m4 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/m4/guestfs-rust.m4 b/m4/guestfs-rust.m4 index aa12a9ef5e5d..1dffd8118874 100644 --- a/m4/guestfs-rust.m4 +++ b/m4/guestfs-rust.m4 @@ -24,8 +24,8 @@ AS_IF([test "x$enable_rust" != "xno"],[ AC_CHECK_PROG([RUSTC],[rustc],[rustc],[no]) AC_CHECK_PROG([CARGO],[cargo],[cargo],[no]) - AS_IF([test "x$RUSTC" == "xno"], [AC_MSG_WARN([rustc not found])]) - AS_IF([test "x$CARGO" == "xno"], [AC_MSG_WARN([cargo not found])]) + AS_IF([test "x$RUSTC" = "xno"], [AC_MSG_WARN([rustc not found])]) + AS_IF([test "x$CARGO" = "xno"], [AC_MSG_WARN([cargo not found])]) ],[ RUSTC=no CARGO=no -- 2.31.1

2 1

libnbd flamegraph Odd benchmark results
by Abhay Raj Singh 30 May '21

30 May '21

Rich, *nbdcopy3-original.svg* is the latest one you sent *nbdcopy-abhay.svg* is the one I was able to create using given commands. However, there was some noise when I compared my results with the original. Is it normal or should I try fixing it. Thanks! Regards, Abhay

2 7

[PATCH libnbd 0/3] Enable multiple connections - take 2
by Nir Soffer 29 May '21

29 May '21

Fix the test using qemu-nbd to allow multiple connections and update the man page about the new semantics. Unfortunately we have no way to tell if a NBD server supports multiple connetions, so our default of using 4 connetions is fragile when using remote qemu-nbd. I think the default should change to use one connection so it always works. Nir Soffer (3): copy: copy-file-to-qcow2.sh: Enable multiple connections Revert "Revert "copy: Always allow multiple connections"" copy: Update nbdcopy(1) about multiple connections copy/copy-file-to-qcow2.sh | 2 +- copy/main.c | 25 ++++++++++++++++--------- copy/nbdcopy.pod | 25 +++++++------------------ 3 files changed, 24 insertions(+), 28 deletions(-) -- 2.26.3

4 6

Few updates
by Frédéric Pierret 28 May '21

28 May '21

Hi, Just few updates that I've given on IRC: For Fedora, since release 33+, the latest DNF update in the kickstart as been commented out. I've tried to add into the kickstart `repo --name=updates` and it looks like the resulting image works and is up to date. For Ubuntu, following https://github.com/rwmjones/guestfs-tools/issues/1, I've used `debian.preseed` partitioning part for Ubuntu and it should solve the issue on the impossibility of resizing the base image. For openSUSE, I've succeeded to revive Leap and notably version 15.2 works great. I hope to be able to send patches soon. Best regards, Frédéric

1 0

[PATCH libnbd v2 0/3] fuse: Enable multithread support
by Richard W.M. Jones 27 May '21

27 May '21

v1 was here: https://listman.redhat.com/archives/libguestfs/2021-May/msg00081.html Compared to v1, patches 1/3 and 3/3 are identical. The new version of patch 2/3 cleans up parallel support. In v1 (https://listman.redhat.com/archives/libguestfs/2021-May/msg00083.html) the thread model was ... interesting. Each FUSE thread would race poll(2) calls on the same file descriptor against each other, and then when one succeeded it would grab a global lock, rerun poll on the file descriptor (to get the true picture of the fd's state), and run the NBD state machine within that global lock but on the FUSE thread. In this version there is a single background thread which dispatches NBD requests on the handle. FUSE threads only start the asynch command, then block waiting for the background thread to finish off their command. Perhaps not too surprisingly v1 is quite a bit faster than v2. v2 suffers from the overhead of coordination between the FUSE threads and the single thread performing the NBD operations (as well as there being only a single thread doing the work). But I think v2 is more maintainable. Multi-conn would help here and also be easier to add with v2. The FIO results presented in the commit messages are stable. Rich.

3 7

[PATCH libnbd] copy: Always allow multiple connections
by Nir Soffer 27 May '21

27 May '21

Using multiple connections when the server does not report can_multi_conn is not safe in general, but for the special access pattern in nbdcopy it is safe. Use multiple connections are used even if the destination (e.g. qemu-nbd) does not report the multi-conn flag. Here is an example run with this change using qemu-nbd and qcow2 images. Using /dev/shm to get stable results. When copying to memory, using 8 in-flight requests and request size of 128k best. I'm comparing also 2m requests to match qemu-img default io size. Testing with images on /dev/shm shows 25% speedup compared with single connection. Testing with FC storage on real images shows 10% speedup[1]. $ qemu-img create -f qcow2 /dev/shm/dst.qcow2 6g $ qemu-nbd -r -t -e 4 -f qcow2 -k /tmp/src.sock /var/tmp/fedora-32.qcow2 & $ qemu-nbd -t -e 4 -f qcow2 -k /tmp/dst.sock /dev/shm/dst.qcow2 & $ SRC=nbd+unix:///?socket=/tmp/src.sock $ DST=nbd+unix:///?socket=/tmp/dst.sock $ hyperfine -w3 -L s 131072,2097152 \ ".libs/nbdcopy --requests=2 --request-size={s} --connections=4 --flush $SRC $DST" \ ".libs/nbdcopy --requests=8 --request-size={s} --connections=1 --flush $SRC $DST" \ "qemu-img convert -n -m 8 -W $SRC $DST" Benchmark 1: .libs/nbdcopy --requests=2 --request-size=131072 --connections=4 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock Time (mean ± σ): 593.8 ms ± 16.6 ms [User: 197.2 ms, System: 506.4 ms] Range (min … max): 567.9 ms … 617.8 ms 10 runs Benchmark 2: .libs/nbdcopy --requests=8 --request-size=131072 --connections=1 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock Time (mean ± σ): 743.4 ms ± 82.4 ms [User: 180.5 ms, System: 497.9 ms] Range (min … max): 631.2 ms … 816.5 ms 10 runs Benchmark 3: qemu-img convert -n -m 8 -W nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock Time (mean ± σ): 1.029 s ± 0.022 s [User: 165.7 ms, System: 655.2 ms] Range (min … max): 0.989 s … 1.067 s 10 runs Benchmark 4: .libs/nbdcopy --requests=2 --request-size=2097152 --connections=4 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock Time (mean ± σ): 1.007 s ± 0.012 s [User: 238.0 ms, System: 961.8 ms] Range (min … max): 0.992 s … 1.029 s 10 runs Benchmark 5: .libs/nbdcopy --requests=8 --request-size=2097152 --connections=1 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock Time (mean ± σ): 1.129 s ± 0.013 s [User: 184.4 ms, System: 743.6 ms] Range (min … max): 1.106 s … 1.149 s 10 runs Summary '.libs/nbdcopy --requests=2 --request-size=131072 --connections=4 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock' ran 1.25 ± 0.14 times faster than '.libs/nbdcopy --requests=8 --request-size=131072 --connections=1 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock' 1.70 ± 0.05 times faster than '.libs/nbdcopy --requests=2 --request-size=2097152 --connections=4 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock' 1.73 ± 0.06 times faster than 'qemu-img convert -n -m 8 -W nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock' 1.90 ± 0.06 times faster than '.libs/nbdcopy --requests=8 --request-size=2097152 --connections=1 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock' [1] https://listman.redhat.com/archives/libguestfs/2021-May/msg00124.html Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- copy/main.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/copy/main.c b/copy/main.c index b9dbe1d..70a4e11 100644 --- a/copy/main.c +++ b/copy/main.c @@ -321,10 +321,6 @@ main (int argc, char *argv[]) exit (EXIT_FAILURE); } - /* If multi-conn is not supported, force connections to 1. */ - if (! src->ops->can_multi_conn (src) || ! dst->ops->can_multi_conn (dst)) - connections = 1; - /* Calculate the number of threads from the number of connections. */ if (threads == 0) { long t; @@ -379,15 +375,26 @@ main (int argc, char *argv[]) synchronous ? "true" : "false"); } - /* If multi-conn is enabled on either side, then at this point we + /* If using multiple connections, then at this point we * need to ask the backend to open the extra connections. + * + * Using multiple connections when the server does not report + * can_multi_conn is not safe in general, but is safe for the special + * access pattern in nbdcopy. + * + * - We split the image to 128m segments, and every segment is written + * by single worker. + * + * - Writes do not overlap, and are never partial block that another + * worker may modify at the same time. + * + * - We never read from the destination so we don't have caching + * synchronization issue between clients. */ if (connections > 1) { assert (threads == connections); - if (src->ops->can_multi_conn (src)) - src->ops->start_multi_conn (src); - if (dst->ops->can_multi_conn (dst)) - dst->ops->start_multi_conn (dst); + src->ops->start_multi_conn (src); + dst->ops->start_multi_conn (dst); } /* If the source is NBD and we couldn't negotiate meta -- 2.26.3

3 4