[PATCH] point users to Libera Chat rather than FreeNode
by Daniel P. Berrangé
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
docs/guestfs-faq.pod | 2 +-
website/index.html.in | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/docs/guestfs-faq.pod b/docs/guestfs-faq.pod
index bea609591..15b1247b0 100644
--- a/docs/guestfs-faq.pod
+++ b/docs/guestfs-faq.pod
@@ -108,7 +108,7 @@ There is a mailing list, mainly for development, but users are also
welcome to ask questions about libguestfs and the virt tools:
L<https://www.redhat.com/mailman/listinfo/libguestfs>
-You can also talk to us on IRC channel C<#libguestfs> on FreeNode.
+You can also talk to us on IRC channel C<#guestfs> on Libera Chat.
We're not always around, so please stay in the channel after asking
your question and someone will get back to you.
diff --git a/website/index.html.in b/website/index.html.in
index f469c5eeb..7453129d6 100644
--- a/website/index.html.in
+++ b/website/index.html.in
@@ -55,8 +55,8 @@ guestfish --ro -i -a disk.img
<p>
Join us on
the <a href="http://www.redhat.com/mailman/listinfo/libguestfs">libguestfs
-mailing list</a>, or on IRC channel <code>#libguestfs</code>
-on <a href="http://freenode.net/">FreeNode</a>.
+mailing list</a>, or on IRC channel <code>#guestfs</code>
+on <a href="https://libera.chat/">Libera Chat</a>.
</p>
</div>
--
2.31.1
3 years, 3 months
Re: [Libguestfs] oVirt import from VMware
by Richard W.M. Jones
[Adding upstream mailing list]
On Mon, May 17, 2021 at 12:32:12PM -0400, Alan Daniels wrote:
> > Thanks Martin. The logs (/var/log/vdsm/import on the host)
> show:
> >
> > qemu-img: /var/tmp/v2vovle5b34c.qcow2: CURL: Error opening
> file:
> > Server does not support 'range' (byte ranges).
>
> This is unexpected, but I'd need to see the full log to be sure.
>
> Given the little information available (but it's vCenter 7 so it would
> fit), it could be:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1846238
> https://bugzilla.redhat.com/show_bug.cgi?id=1841038
>
> (They're the same bug, we didn't fix it in RHEL 7)
> Hey Rich,
>
> Thanks for the help! Some more info about the environment:
>
> Oracle Linux 7.9
> # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.9 (Maipo)
>
> OLVM / oVirt 4.3
>
> virt-v2v.x86_64 1:1.40.2-10.0.1.el7
> libguestfs.x86_64 1:1.40.2-10.0.1.el7
> qemu-kvm.x86_64 15:4.2.1-6.el7
> nbdkit.x86_64 1.8.0-4.el7
> qemu-block-curl.x86_64 15:4.2.1-6.el7
>
> Running on Oracle Cloud.
>
> VMware is on another cloud provider and they're connected via VPN.
>
> The logs looks essentially the same as the one from https://bugzilla.redhat.com
> /show_bug.cgi?id=1846238
>
> I can't really upgrade from 7.9 to 8.3, so is this expected behavior? Are there
> some potential workarounds?
It's a bug in qemu which you will need to patch in order to do the
import using this method.
Other methods don't suffer from the bug because they don't use qemu in
the same way. I would recommend exploring the “-i ova” method.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
3 years, 5 months
[PATCH v2v v3 REBASE] rhv-upload: Validate UUIDs passed to -oo rhv-disk-uuid (RHBZ#1789279)
by Martin Kletzander
The validation helps us fail early and with a sensible error message. The NIL
UUID is not valid for oVirt, but other than that there is no other logic in
there merely because the UUID types are a matter of the generator and they are
just forwarded in this partucular case.
On top of that also check that the UUID is not already taken. This makes the
code fail with a sensible error message instead of cryptic error from ovirtsdk.
Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com>
---
v4:
- https://listman.redhat.com/archives/libguestfs/2021-May/msg00095.html
- Merge the two patches together
- Things mentioned in the review are already in, so there is no other change
v3:
- https://listman.redhat.com/archives/libguestfs/2020-March/msg00084.html
- Do the check in precheck
- Fix for Lazy evaluation of regexp UUID
v2:
- https://www.redhat.com/archives/libguestfs/2020-January/msg00221.html
- Use EEXIST instead of EINVAL
- Put the colliding UUID into the error
- Do not evaluate the PCRE needlessly multiple times
v1:
- https://www.redhat.com/archives/libguestfs/2020-January/msg00184.html
v2v/output_rhv_upload.ml | 18 ++++++++++++++++++
v2v/rhv-upload-precheck.py | 10 ++++++++++
2 files changed, 28 insertions(+)
diff --git a/v2v/output_rhv_upload.ml b/v2v/output_rhv_upload.ml
index dbef7011b04b..15ba1078019e 100644
--- a/v2v/output_rhv_upload.ml
+++ b/v2v/output_rhv_upload.ml
@@ -49,6 +49,16 @@ after their uploads (if you do, you must supply one for each disk):
-oo rhv-disk-uuid=UUID Disk UUID
")
+let is_nonnil_uuid uuid =
+ let nil_uuid = "00000000-0000-0000-0000-000000000000" in
+ let rex_uuid = lazy (
+ let hex = "[a-fA-F0-9]" in
+ let str = sprintf "^%s{8}-%s{4}-%s{4}-%s{4}-%s{12}$" hex hex hex hex hex in
+ PCRE.compile str
+ ) in
+ if uuid = nil_uuid then false
+ else PCRE.matches (Lazy.force rex_uuid) uuid
+
let parse_output_options options =
let rhv_cafile = ref None in
let rhv_cluster = ref None in
@@ -71,6 +81,8 @@ let parse_output_options options =
| "rhv-verifypeer", "" -> rhv_verifypeer := true
| "rhv-verifypeer", v -> rhv_verifypeer := bool_of_string v
| "rhv-disk-uuid", v ->
+ if not (is_nonnil_uuid v) then
+ error (f_"-o rhv-upload: invalid UUID for -oo rhv-disk-uuid");
rhv_disk_uuids := Some (v :: (Option.default [] !rhv_disk_uuids))
| k, _ ->
error (f_"-o rhv-upload: unknown output option ‘-oo %s’") k
@@ -256,6 +268,12 @@ object
error_unless_output_alloc_sparse output_alloc;
(* Python code prechecks. *)
+ let json_params = match rhv_options.rhv_disk_uuids with
+ | None -> json_params
+ | Some uuids ->
+ let ids = List.map (fun uuid -> JSON.String uuid) uuids in
+ ("rhv_disk_uuids", JSON.List ids) :: json_params
+ in
let precheck_fn = tmpdir // "v2vprecheck.json" in
let fd = Unix.openfile precheck_fn [O_WRONLY; O_CREAT] 0o600 in
if Python_script.run_command ~stdout_fd:fd
diff --git a/v2v/rhv-upload-precheck.py b/v2v/rhv-upload-precheck.py
index 2180ea1043ab..c3f74df35f65 100644
--- a/v2v/rhv-upload-precheck.py
+++ b/v2v/rhv-upload-precheck.py
@@ -97,6 +97,16 @@ if cpu.architecture == types.Architecture.UNDEFINED:
raise RuntimeError("The cluster ‘%s’ has an unknown architecture" %
(params['rhv_cluster']))
+# Find if any disk already exists with specified UUID.
+disks_service = system_service.disks_service()
+
+for uuid in params.get('rhv_disk_uuids', []):
+ try:
+ disk_service = disks_service.disk_service(uuid).get()
+ raise RuntimeError("Disk with the UUID '%s' already exists" % uuid)
+ except sdk.NotFoundError:
+ pass
+
# Otherwise everything is OK, print a JSON with the results.
results = {
"rhv_storagedomain_uuid": storage_domain.id,
--
2.31.1
3 years, 5 months
[PATCH] build: Don't use non-POSIX tests
by Martin Kletzander
The `test` builtin/binary usually accepts `==` for string comparison, it is
mostly accepted for typos and people being used to double equals, but is not
documented and not always accepted either. Since autoconf uses the default
shell, it might just fail in some cases with:
./configure: 29986: test: xrustc: unexpected operator
./configure: 29990: test: xcargo: unexpected operator
Just change it to single equals as it is done everywhere else.
Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com>
---
m4/guestfs-rust.m4 | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/m4/guestfs-rust.m4 b/m4/guestfs-rust.m4
index aa12a9ef5e5d..1dffd8118874 100644
--- a/m4/guestfs-rust.m4
+++ b/m4/guestfs-rust.m4
@@ -24,8 +24,8 @@ AS_IF([test "x$enable_rust" != "xno"],[
AC_CHECK_PROG([RUSTC],[rustc],[rustc],[no])
AC_CHECK_PROG([CARGO],[cargo],[cargo],[no])
- AS_IF([test "x$RUSTC" == "xno"], [AC_MSG_WARN([rustc not found])])
- AS_IF([test "x$CARGO" == "xno"], [AC_MSG_WARN([cargo not found])])
+ AS_IF([test "x$RUSTC" = "xno"], [AC_MSG_WARN([rustc not found])])
+ AS_IF([test "x$CARGO" = "xno"], [AC_MSG_WARN([cargo not found])])
],[
RUSTC=no
CARGO=no
--
2.31.1
3 years, 5 months
libnbd flamegraph Odd benchmark results
by Abhay Raj Singh
Rich, *nbdcopy3-original.svg* is the latest one you sent
*nbdcopy-abhay.svg* is the one I was able to create using given commands.
However, there was some noise when I compared my results with the original.
Is it normal or should I try fixing it. Thanks!
Regards,
Abhay
3 years, 5 months
[PATCH libnbd 0/3] Enable multiple connections - take 2
by Nir Soffer
Fix the test using qemu-nbd to allow multiple connections and update the
man page about the new semantics.
Unfortunately we have no way to tell if a NBD server supports multiple
connetions, so our default of using 4 connetions is fragile when using
remote qemu-nbd. I think the default should change to use one connection
so it always works.
Nir Soffer (3):
copy: copy-file-to-qcow2.sh: Enable multiple connections
Revert "Revert "copy: Always allow multiple connections""
copy: Update nbdcopy(1) about multiple connections
copy/copy-file-to-qcow2.sh | 2 +-
copy/main.c | 25 ++++++++++++++++---------
copy/nbdcopy.pod | 25 +++++++------------------
3 files changed, 24 insertions(+), 28 deletions(-)
--
2.26.3
3 years, 5 months
Few updates
by Frédéric Pierret
Hi,
Just few updates that I've given on IRC:
For Fedora, since release 33+, the latest DNF update in the kickstart as been commented out. I've tried to add into the kickstart `repo --name=updates` and it looks like the resulting image works and is up to date.
For Ubuntu, following https://github.com/rwmjones/guestfs-tools/issues/1, I've used `debian.preseed` partitioning part for Ubuntu and it should solve the issue on the impossibility of resizing the base image.
For openSUSE, I've succeeded to revive Leap and notably version 15.2 works great.
I hope to be able to send patches soon.
Best regards,
Frédéric
3 years, 6 months
[PATCH libnbd v2 0/3] fuse: Enable multithread support
by Richard W.M. Jones
v1 was here:
https://listman.redhat.com/archives/libguestfs/2021-May/msg00081.html
Compared to v1, patches 1/3 and 3/3 are identical.
The new version of patch 2/3 cleans up parallel support. In v1
(https://listman.redhat.com/archives/libguestfs/2021-May/msg00083.html)
the thread model was ... interesting. Each FUSE thread would race
poll(2) calls on the same file descriptor against each other, and then
when one succeeded it would grab a global lock, rerun poll on the file
descriptor (to get the true picture of the fd's state), and run the
NBD state machine within that global lock but on the FUSE thread.
In this version there is a single background thread which dispatches
NBD requests on the handle. FUSE threads only start the asynch
command, then block waiting for the background thread to finish off
their command.
Perhaps not too surprisingly v1 is quite a bit faster than v2. v2
suffers from the overhead of coordination between the FUSE threads and
the single thread performing the NBD operations (as well as there
being only a single thread doing the work). But I think v2 is more
maintainable. Multi-conn would help here and also be easier to add
with v2.
The FIO results presented in the commit messages are stable.
Rich.
3 years, 6 months
[PATCH libnbd] copy: Always allow multiple connections
by Nir Soffer
Using multiple connections when the server does not report
can_multi_conn is not safe in general, but for the special access
pattern in nbdcopy it is safe.
Use multiple connections are used even if the destination (e.g.
qemu-nbd) does not report the multi-conn flag.
Here is an example run with this change using qemu-nbd and qcow2
images. Using /dev/shm to get stable results.
When copying to memory, using 8 in-flight requests and request size of
128k best. I'm comparing also 2m requests to match qemu-img default io
size.
Testing with images on /dev/shm shows 25% speedup compared with single
connection. Testing with FC storage on real images shows 10% speedup[1].
$ qemu-img create -f qcow2 /dev/shm/dst.qcow2 6g
$ qemu-nbd -r -t -e 4 -f qcow2 -k /tmp/src.sock /var/tmp/fedora-32.qcow2 &
$ qemu-nbd -t -e 4 -f qcow2 -k /tmp/dst.sock /dev/shm/dst.qcow2 &
$ SRC=nbd+unix:///?socket=/tmp/src.sock
$ DST=nbd+unix:///?socket=/tmp/dst.sock
$ hyperfine -w3 -L s 131072,2097152 \
".libs/nbdcopy --requests=2 --request-size={s} --connections=4 --flush $SRC $DST" \
".libs/nbdcopy --requests=8 --request-size={s} --connections=1 --flush $SRC $DST" \
"qemu-img convert -n -m 8 -W $SRC $DST"
Benchmark 1: .libs/nbdcopy --requests=2 --request-size=131072 --connections=4 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 593.8 ms ± 16.6 ms [User: 197.2 ms, System: 506.4 ms]
Range (min … max): 567.9 ms … 617.8 ms 10 runs
Benchmark 2: .libs/nbdcopy --requests=8 --request-size=131072 --connections=1 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 743.4 ms ± 82.4 ms [User: 180.5 ms, System: 497.9 ms]
Range (min … max): 631.2 ms … 816.5 ms 10 runs
Benchmark 3: qemu-img convert -n -m 8 -W nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 1.029 s ± 0.022 s [User: 165.7 ms, System: 655.2 ms]
Range (min … max): 0.989 s … 1.067 s 10 runs
Benchmark 4: .libs/nbdcopy --requests=2 --request-size=2097152 --connections=4 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 1.007 s ± 0.012 s [User: 238.0 ms, System: 961.8 ms]
Range (min … max): 0.992 s … 1.029 s 10 runs
Benchmark 5: .libs/nbdcopy --requests=8 --request-size=2097152 --connections=1 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 1.129 s ± 0.013 s [User: 184.4 ms, System: 743.6 ms]
Range (min … max): 1.106 s … 1.149 s 10 runs
Summary
'.libs/nbdcopy --requests=2 --request-size=131072 --connections=4 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock' ran
1.25 ± 0.14 times faster than '.libs/nbdcopy --requests=8 --request-size=131072 --connections=1 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
1.70 ± 0.05 times faster than '.libs/nbdcopy --requests=2 --request-size=2097152 --connections=4 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
1.73 ± 0.06 times faster than 'qemu-img convert -n -m 8 -W nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
1.90 ± 0.06 times faster than '.libs/nbdcopy --requests=8 --request-size=2097152 --connections=1 --flush nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
[1] https://listman.redhat.com/archives/libguestfs/2021-May/msg00124.html
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
copy/main.c | 25 ++++++++++++++++---------
1 file changed, 16 insertions(+), 9 deletions(-)
diff --git a/copy/main.c b/copy/main.c
index b9dbe1d..70a4e11 100644
--- a/copy/main.c
+++ b/copy/main.c
@@ -321,10 +321,6 @@ main (int argc, char *argv[])
exit (EXIT_FAILURE);
}
- /* If multi-conn is not supported, force connections to 1. */
- if (! src->ops->can_multi_conn (src) || ! dst->ops->can_multi_conn (dst))
- connections = 1;
-
/* Calculate the number of threads from the number of connections. */
if (threads == 0) {
long t;
@@ -379,15 +375,26 @@ main (int argc, char *argv[])
synchronous ? "true" : "false");
}
- /* If multi-conn is enabled on either side, then at this point we
+ /* If using multiple connections, then at this point we
* need to ask the backend to open the extra connections.
+ *
+ * Using multiple connections when the server does not report
+ * can_multi_conn is not safe in general, but is safe for the special
+ * access pattern in nbdcopy.
+ *
+ * - We split the image to 128m segments, and every segment is written
+ * by single worker.
+ *
+ * - Writes do not overlap, and are never partial block that another
+ * worker may modify at the same time.
+ *
+ * - We never read from the destination so we don't have caching
+ * synchronization issue between clients.
*/
if (connections > 1) {
assert (threads == connections);
- if (src->ops->can_multi_conn (src))
- src->ops->start_multi_conn (src);
- if (dst->ops->can_multi_conn (dst))
- dst->ops->start_multi_conn (dst);
+ src->ops->start_multi_conn (src);
+ dst->ops->start_multi_conn (dst);
}
/* If the source is NBD and we couldn't negotiate meta
--
2.26.3
3 years, 6 months
