March 2020 - Libguestfs - libguestfs List Archives

Re: [Libguestfs] Use libguestfs with SAS URI of a disk image
by Richard W.M. Jones 16 Mar '20

16 Mar '20

On Sat, Mar 14, 2020 at 04:48:24AM +0000, JP Zhang wrote: > HI Richard, > > Thanks for the quick reply. > > The Python program that works, that I mentioned earlier, uses subprocess to issue/run "guestfish" commands below in that order. > /libguestfs/run guestfish --listen -a <https storage-SAS-URL> -- ro > then the Python code exported $PID to the environment. I believe so that the same $PID could be used for the subsequent guestfish calls. > /libguestfs/run guestfish --remote -- -version > /libguestfs/run guestfish --remote launch > > Since it is running in the subprocess of "run.sh" shell, I couldn't see the libguestfs debug output. Maybe there is a way, I just don't know. > > Then I manually typed similar guestfish commands in Linux bash shell, like > guestfish --listen -a <https storage-SAS-URL> -- ro > <this returned a $PID> > guestfish --remote=<$PID> -- -version > guestfish --remote=<$PID> launch > this sequence of guestfish commands failed at "launch" cmd. > > regarding params passed to add_drive call in libguestfs, the debug trace from running the guestfish cmds above looks like this: > libguestfs: trace: add_drive "/<container>/abcd" "protocol:https" "server:tcp:<account>.blob.core.windows.net" > which is something I already tried, not any more insights. You'll need the trace from the working call, not the call that doesn't work. Try ‘export LIBGUESTFS_TRACE=1’ before the working command. > I really could not tell the difference between these two set of guestfish commands. > > (1) do you see any substance difference between doing things in these two sets of commands? ones issued in Python program and ones issued in Bash shell manually? > (2) if I can add a local VM disk image to libguestfs and list its files, can I safely say that I have libguestfs installed or set up correctly on my machine? meaning I should be able to do "network" based disk image calls/add "network" based disk image? correct, in theory? At this point, I cannot 100% trust my local linguestfs setup since I don't know much about libguestfs. > (3) BTW, I see error message in manual run in Bash shell above. does this tell us anything? > Could not access KVM kernel module: No such file or directory > qemu-system-x86_64: failed to initialize KVM: No such file or directory > qemu-system-x86_64: Back to tcg accelerator > I see the same KVM error when I ran "libguestfs-test-tool" > Is KVM kernel absolutely needed? or is this just an option to try and does libguestfs fall back to other Linux kernel to build supermin appliance > > BTW, the working Python program was developed by others whom we have no contact now. So far I don't see anything special about Python. It is a just a shell program to issue libguestfs commands and wire up our own processing logic on top of libguestfs. > > Regarding using "nbdkit", it looks like it is another service I need to run on my box. The goal of my project is to reduce the hops and improve the performance. Also the code I develop will be a web service that takes many real time requests. with nbdkit, does it mean that I need to create a instance of nbdkit on the fly for every request that my service receives? will nbdkit option work well when I have multiple concurrent libguestfs calls going through nbdkit in multiple threads/processes? Actually nbdkit doesn't need to be run as a separate service. You can run it "captive" from the main program. It also has several advantages over using libguestfs to connect to curl directly. You probably want to look at what we do in virt-v2v, where we have almost exactly the same scenario and we use nbdkit to proxy the connection: https://github.com/libguestfs/virt-v2v/blob/master/v2v/nbdkit.ml https://github.com/libguestfs/virt-v2v/blob/f8e4e8643f105bc2abeb6b80b17a0de… Also at: http://libguestfs.org/nbdkit.1.html http://libguestfs.org/nbdkit-captive.1.html http://libguestfs.org/nbdkit-curl-plugin.1.html http://libguestfs.org/nbdkit-cache-filter.1.html Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

2 1

[PATCH v3 00/34] Configurable policy for handling deprecated interfaces
by Markus Armbruster 15 Mar '20

15 Mar '20

This series extends QMP introspection to cover deprecation. Additionally, new option -compat lets you configure what to do when deprecated interfaces get used. This is intended for testing users of the management interfaces. It is experimental. -compat deprecated-input=<in-policy> configures what to do when deprecated input is received. Available policies: * accept: Accept deprecated commands and arguments (default) * reject: Reject them * crash: Crash -compat deprecated-output=<out-policy> configures what to do when deprecated output is sent. Available output policies: * accept: Emit deprecated command results and events (default) * hide: Suppress them For now, -compat covers only deprecated syntactic aspects of QMP. We may want to extend it to cover semantic aspects, CLI, and experimental features. PATCH 01-04: Documentation fixes PATCH 05-10: Test improvements PATCH 11-24: Add feature flags to remaining user-defined types and to struct members PATCH 25-26: New special feature 'deprecated', visible in introspection PATCH 27-34: New -compat to set policy for handling stuff marked with feature 'deprecated' v3: * Rebased, non-trivial conflicts in PATCH 01+26+27+34 due to RST conversion and code motion * PATCH 28-29: Old PATCH 28 split up to ease review * PATCH 30-31: New * PATCH 32-33: Old PATCH 29 split up to ease review Comparison to RFC (24 Oct 2019): * Cover arguments and results in addition to commands and events * Half-baked "[RFC PATCH 18/19] qapi: Include a warning in the response to a deprecated command" dropped See also last item of Subject: Minutes of KVM Forum BoF on deprecating stuff Date: Fri, 26 Oct 2018 16:03:51 +0200 Message-ID: <87mur0ls8o.fsf(a)dusky.pond.sub.org> https://lists.nongnu.org/archive/html/qemu-devel/2018-10/msg05828.html Cc: Lukáš Doktor <ldoktor(a)redhat.com> Cc: libguestfs(a)redhat.com Cc: libvir-list(a)redhat.com Cc: Daniel P. Berrange <berrange(a)redhat.com> Cc: Peter Krempa <pkrempa(a)redhat.com> Markus Armbruster (34): qemu-doc: Belatedly document QMP command arg & result deprecation qapi: Belatedly update doc comment for @wait deprecation docs/devel/qapi-code-gen: Clarify allow-oob introspection docs/devel/qapi-code-gen: Document 'features' introspection tests/test-qmp-cmds: Factor out qmp_dispatch() test helpers tests/test-qmp-cmds: Check responses more thoroughly tests/test-qmp-cmds: Simplify test data setup tests/test-qmp-event: Simplify test data setup tests/test-qmp-event: Use qobject_is_equal() tests/test-qmp-event: Check event is actually emitted qapi/schema: Clean up around QAPISchemaEntity.connect_doc() qapi: Add feature flags to remaining definitions qapi: Consistently put @features parameter right after @ifcond qapi/introspect: Rename *qlit* to reduce confusion qapi/introspect: Factor out _make_tree() qapi/schema: Change _make_features() to a take feature list qapi/schema: Reorder classes so related ones are together qapi/schema: Rename QAPISchemaObjectType{Variant,Variants} qapi/schema: Call QAPIDoc.connect_member() in just one place qapi: Add feature flags to struct members qapi: Inline do_qmp_dispatch() into qmp_dispatch() qapi: Simplify how qmp_dispatch() deals with QCO_NO_SUCCESS_RESP qapi: Simplify how qmp_dispatch() gets the request ID qapi: Replace qmp_dispatch()'s TODO comment by an explanation qapi: New special feature flag "deprecated" qapi: Mark deprecated QMP parts with feature 'deprecated' qemu-options: New -compat to set policy for deprecated interfaces qapi: Implement deprecated-output=hide for QMP command results qapi: Implement deprecated-output=hide for QMP events qapi: Implement deprecated-output=hide for QMP event data qapi: Implement deprecated-output=hide for QMP introspection qapi: Implement deprecated-input=reject for QMP commands qapi: Implement deprecated-input=reject for QMP command arguments qapi: New -compat deprecated-input=crash docs/devel/qapi-code-gen.txt | 79 ++- docs/system/deprecated.rst | 48 +- tests/qapi-schema/doc-good.texi | 32 ++ qapi/block-core.json | 48 +- qapi/block.json | 30 +- qapi/char.json | 1 + qapi/compat.json | 52 ++ qapi/control.json | 11 +- qapi/introspect.json | 28 +- qapi/machine.json | 34 +- qapi/migration.json | 36 +- qapi/misc.json | 13 +- qapi/qapi-schema.json | 1 + include/qapi/compat-policy.h | 20 + include/qapi/qmp/dispatch.h | 1 + include/qapi/qobject-input-visitor.h | 9 + include/qapi/qobject-output-visitor.h | 9 + include/qapi/visitor-impl.h | 3 + include/qapi/visitor.h | 9 + monitor/monitor-internal.h | 3 - monitor/misc.c | 2 - monitor/qmp-cmds-control.c | 102 +++- qapi/qapi-visit-core.c | 9 + qapi/qmp-dispatch.c | 137 ++--- qapi/qobject-input-visitor.c | 29 ++ qapi/qobject-output-visitor.c | 20 + qemu-storage-daemon.c | 2 - softmmu/vl.c | 17 + tests/test-qmp-cmds.c | 249 +++++---- tests/test-qmp-event.c | 201 +++----- qapi/Makefile.objs | 8 +- qapi/trace-events | 1 + qemu-options.hx | 22 + scripts/qapi/commands.py | 20 +- scripts/qapi/doc.py | 16 +- scripts/qapi/events.py | 22 +- scripts/qapi/expr.py | 14 +- scripts/qapi/introspect.py | 104 ++-- scripts/qapi/schema.py | 488 ++++++++++-------- scripts/qapi/types.py | 8 +- scripts/qapi/visit.py | 28 +- tests/Makefile.include | 1 + tests/qapi-schema/alternate-base.err | 2 +- tests/qapi-schema/doc-good.json | 22 +- tests/qapi-schema/doc-good.out | 18 + .../qapi-schema/features-deprecated-type.err | 2 + .../qapi-schema/features-deprecated-type.json | 3 + .../qapi-schema/features-deprecated-type.out | 0 tests/qapi-schema/qapi-schema-test.json | 51 +- tests/qapi-schema/qapi-schema-test.out | 48 +- tests/qapi-schema/test-qapi.py | 26 +- 51 files changed, 1384 insertions(+), 755 deletions(-) create mode 100644 qapi/compat.json create mode 100644 include/qapi/compat-policy.h create mode 100644 tests/qapi-schema/features-deprecated-type.err create mode 100644 tests/qapi-schema/features-deprecated-type.json create mode 100644 tests/qapi-schema/features-deprecated-type.out -- 2.21.1

1 0

Re: [Libguestfs] Use libguestfs with SAS URI of a disk image
by Richard W.M. Jones 13 Mar '20

13 Mar '20

On Fri, Mar 13, 2020 at 09:41:37PM +0000, JP Zhang wrote: > Hi Richard, > > Thanks for the great work you did on libguestfs. > > I try to call libguestfs lib API directly in C to "add drive" by > using a Azure blob storage access SAS URI, which has shared access > token/secret on the URL. > > The URL likes like this: > 'https://<account>.blob.core.windows.net/<container>/<VMDiskImage>?sv=2017-04-17&sr=b&si=<guid>&sig=<secret>' > I try to use this API: > > int > guestfs_add_drive_opts (guestfs_h *g, > const char *filename, > ...); > > > My code looks like this: > char const *servers[] = { (char const *)"<account>.blob.core.windows.net", NULL }; > int addDriveRet = guestfs_add_drive_opts (g, "/<container>/<VMDiskImage>?sv=2017-04-17&sr=b&si=<guid>&sig=<secret>", > GUESTFS_ADD_DRIVE_OPTS_FORMAT, "raw", > GUESTFS_ADD_DRIVE_OPTS_PROTOCOL, "https", > GUESTFS_ADD_DRIVE_OPTS_SERVER, servers, > GUESTFS_ADD_DRIVE_OPTS_READONLY, 1, > -1); Since you say you got it to work with guestfish, one trick is to run ‘guestfish -x’ (or the ‘trace true’ command, or set LIBGUESTFS_TRACE=1) and it will actually print the full API call you should use. I believe the problem may be to do with URI %-quoting of the filename (2nd) parameter. > I read the documentation and samples on libguestfs.org in depth. I > could not find a good sample for "https" with SAS secret on URL. I > tried many different ways to break up the URL into "server", > "protocol", and secret. It looks like "secret" option is only for > "rbd". From the debug log/printouts from libguestfs, it seemed to > reassemble the parts of original URL back to a URL that matched the > original one. But I got http 403 error( auth error). I am very sure > the URL is correct because I can download the disk image using > Curl. The libguestfs debug log printout displayed the URL with ? > being turned into %3F and % into %25. I am not sure whether > libguestfs URL-encoded printout was just for printout or whether it > was the actual URL string being used to make the https server > call. If it was, then the secret has been changed due to URL > encoding again. This is just a guess. Maybe not related at all. > > BTW, I can run libguestfs-test-tool successfully. and I have another > program in Python that call through /run.sh -> GuestFish -> > libguestfs lib, it works with the same original SAS URL. So you can contact the SAS URL through this? If so I would try the trace trick above to see the actual protocol call being made. If that doesn't work, another way to do it would be to forward the connection through nbdkit, ie: rm -f /tmp/socket nbdkit -f -U /tmp/socket curl 'https://...' & and connect to the the NBD socket using the add_drive_opts API as documented in the manual. (in guestfish you'd use: guestfish -a 'nbd://?socket=/tmp/socket') nbdkit can also be run captive from your program, see http://libguestfs.org/nbdkit-captive.1.html > also, on my dev machine, I can do "guestfs_add_drive" on the local > disk image successfully and mount the device, and list files after I > downloaded the same disk image to my local machine. > > guestfs_add_drive(g, "/pathto/vmdiskimage"); > > Could you share how I should fill in various params in > "guest_add_drive_opts()? like "PROTOCOL", "SERVER", "filename"( the > 2nd param), or USERNAME ( not seem to apply here, I also tried with > putting "secret" in SECRET, "sig" in USERNAME, the rebuilt URL from > libguestfs looks missing these parts in final URL, it probably put > these in HTTP header.) > > I also try to call guestfs_add_drive() with the original URL > below. it didn't work either. > > guestfs_add_drive(g, "https://<account>.blob.core.windows.net/<container>/<VMDiskImage>?sv=2017-04-17&sr=b&si=<guid>&sig=<secret>"); Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

1 0

[libnbd PATCH] lib: remove extra @LIBS@ from pkg-config file
by Pino Toscano 12 Mar '20

12 Mar '20

At the moment it is empty, so probably it does not exist. Remove it to avoid adding spurious content to the pkg-config file in case that variable will get a value in the future. --- lib/libnbd.pc.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/libnbd.pc.in b/lib/libnbd.pc.in index ba91f49..294c090 100644 --- a/lib/libnbd.pc.in +++ b/lib/libnbd.pc.in @@ -8,4 +8,4 @@ Version: @PACKAGE_VERSION@ Description: NBD client library in userspace Requires: Cflags: -Libs: -lnbd @LIBS@ +Libs: -lnbd -- 2.24.1

2 2

[PATCH] get-kernel: fallback to uncompressed vmlinux kernels
by Pino Toscano 12 Mar '20

12 Mar '20

In case the Linux guest has no compressed linux kernel (vmlinuz), fallback to uncompressed ones (vmlinux). Reported by: Laurent Vivier. --- get-kernel/get_kernel.ml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/get-kernel/get_kernel.ml b/get-kernel/get_kernel.ml index 3015c0fca..e485cf495 100644 --- a/get-kernel/get_kernel.ml +++ b/get-kernel/get_kernel.ml @@ -144,6 +144,9 @@ and pick_kernel_files_linux (g : Guestfs.guestfs) root = let kernels = glob "/boot/vmlinuz-*" in let initrds = glob "/boot/initramfs-*" in + (* Uncompressed kernels: *) + let kernels = if kernels <> [] then kernels else glob "/boot/vmlinux-*" in + (* Old RHEL: *) let initrds = if initrds <> [] then initrds else glob "/boot/initrd-*" in -- 2.24.1

2 1

[hivex PATCH] Remove extra @LIBS@ from pkg-config file
by Pino Toscano 12 Mar '20

12 Mar '20

At the moment it is empty, so probably it does not exist. Remove it to avoid adding spurious content to the pkg-config file in case that variable will get a value in the future. --- hivex.pc.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hivex.pc.in b/hivex.pc.in index c5f594a..250a58b 100644 --- a/hivex.pc.in +++ b/hivex.pc.in @@ -8,4 +8,4 @@ Version: @VERSION@ Description: Read and write Windows Registry Hive files. Requires: Cflags: -Libs: -lhivex @LIBS@ +Libs: -lhivex -- 2.24.1

2 1

[PATCH v2v v3 0/2] rhv-upload: Validate UUIDs and check they don't exist already
by Martin Kletzander 12 Mar '20

12 Mar '20

My stab v3 at fixing this: https://bugzilla.redhat.com/show_bug.cgi?id=1789279 It took me quite some time to go through the whole rfc 4122 just to realize we do not need to do anything with the versions. v3: - Do the check in precheck - Fix for Lazy evaluation of regexp UUID v2: - https://www.redhat.com/archives/libguestfs/2020-January/msg00221.html - Use EEXIST instead of EINVAL - Put the colliding UUID into the error - Do not evaluate the PCRE needlessly multiple times v1: https://www.redhat.com/archives/libguestfs/2020-January/msg00184.html Martin Kletzander (2): rhv-upload: Validate UUIDs passed to -oo rhv-disk-uuid (RHBZ#1789279) rhv-upload: Check that rhv-disk-uuid is not already taken (RHBZ#1789279) v2v/output_rhv_upload.ml | 18 ++++++++++++++++++ v2v/rhv-upload-precheck.py | 10 ++++++++++ 2 files changed, 28 insertions(+) -- 2.25.1

2 4

[PATCH virt-v2v] v2v: Use LIBGUESTFS_CFLAGS/LIBS when compiling and linking.
by Richard W.M. Jones 11 Mar '20

11 Mar '20

Allows virt-v2v to be compiled against the libguestfs build directory. --- v2v/Makefile.am | 2 ++ 1 file changed, 2 insertions(+) diff --git a/v2v/Makefile.am b/v2v/Makefile.am index 2a196d45d..104420d13 100644 --- a/v2v/Makefile.am +++ b/v2v/Makefile.am @@ -197,6 +197,7 @@ virt_v2v_CPPFLAGS = \ -I$(top_srcdir)/lib virt_v2v_CFLAGS = \ $(WARN_CFLAGS) $(WERROR_CFLAGS) \ + $(LIBGUESTFS_CFLAGS) \ $(LIBVIRT_CFLAGS) \ $(LIBOSINFO_CFLAGS) @@ -230,6 +231,7 @@ endif OCAMLCLIBS = \ -lqemuopts \ + $(LIBGUESTFS_LIBS) \ $(LIBVIRT_LIBS) \ $(LIBXML2_LIBS) \ $(JANSSON_LIBS) \ -- 2.24.1

2 2

[PATCH] lib: remove extra @LIBS@ from pkg-config file
by Pino Toscano 11 Mar '20

11 Mar '20

At the moment it is empty, so probably it does not exist. Remove it to avoid adding spurious content to the pkg-config file in case that variable will get a value in the future. --- lib/libguestfs.pc.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/libguestfs.pc.in b/lib/libguestfs.pc.in index 679ed7eba..013bf0f28 100644 --- a/lib/libguestfs.pc.in +++ b/lib/libguestfs.pc.in @@ -8,4 +8,4 @@ Version: @VERSION@ Description: libguestfs library for accessing and modifying VM images Requires: Cflags: -Libs: -lguestfs @LIBS@ +Libs: -lguestfs -- 2.24.1

3 4

[PATCH] builder: templates: add the AppStream repo
by Pino Toscano 11 Mar '20

11 Mar '20

It is generally used together with BaseOS (which contains a restricted set of packages), so add it to the preconfigured repositories by default. --- builder/templates/make-template.ml | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/builder/templates/make-template.ml b/builder/templates/make-template.ml index 8d26177bf..a4e2f37fb 100755 --- a/builder/templates/make-template.ml +++ b/builder/templates/make-template.ml @@ -1238,7 +1238,8 @@ and make_rhel_yum_conf major minor arch = major major minor in sprintf "%s/Server/%s/os" topurl arch, sprintf "%s/source/SRPMS" topurl, - Some (sprintf "%s/Server/optional/%s/os" arch topurl, + Some ("Optional", + sprintf "%s/Server/optional/%s/os" arch topurl, sprintf "%s/Server/optional/source/SRPMS" topurl) | 7, (X86_64|PPC64|PPC64le|S390X) -> let topurl = @@ -1246,7 +1247,8 @@ and make_rhel_yum_conf major minor arch = major major minor in sprintf "%s/Server/%s/os" topurl (string_of_arch arch), sprintf "%s/Server/source/tree" topurl, - Some (sprintf "%s/Server-optional/%s/os" topurl (string_of_arch arch), + Some ("Optional", + sprintf "%s/Server-optional/%s/os" topurl (string_of_arch arch), sprintf "%s/Server-optional/source/tree" topurl) | 7, Aarch64 -> let topurl = @@ -1254,7 +1256,8 @@ and make_rhel_yum_conf major minor arch = major major minor in sprintf "%s/Server/%s/os" topurl (string_of_arch arch), sprintf "%s/Server/source/tree" topurl, - Some (sprintf "%s/Server-optional/%s/os" topurl (string_of_arch arch), + Some ("Optional", + sprintf "%s/Server-optional/%s/os" topurl (string_of_arch arch), sprintf "%s/Server-optional/source/tree" topurl) | 8, arch -> let topurl = @@ -1262,7 +1265,9 @@ and make_rhel_yum_conf major minor arch = major major minor in sprintf "%s/BaseOS/%s/os" topurl (string_of_arch arch), sprintf "%s/BaseOS/source/tree" topurl, - None (* XXX sort out AppStream and CRB *) + Some ("AppStream", + sprintf "%s/AppStream/%s/os" topurl (string_of_arch arch), + sprintf "%s/AppStream/source/tree" topurl) | _ -> assert false in bpf "\ @@ -1285,23 +1290,25 @@ keepcache=0 (match optional with | None -> () - | Some (optionalbaseurl, optionalsrpms) -> + | Some (name, optionalbaseurl, optionalsrpms) -> + let lc_name = String.lowercase_ascii name in bpf "\ -[rhel%d-optional] -name=RHEL %d Server Optional +[rhel%d-%s] +name=RHEL %d Server %s baseurl=%s enabled=1 gpgcheck=0 keepcache=0 -[rhel%d-optional-source] -name=RHEL %d Server Optional +[rhel%d-%s-source] +name=RHEL %d Server %s baseurl=%s enabled=0 gpgcheck=0 keepcache=0 -" major major optionalbaseurl major major optionalsrpms +" major lc_name major lc_name optionalbaseurl + major lc_name major lc_name optionalsrpms ) ) else ( assert false (* not implemented for RHEL major >= 9 *) -- 2.24.1

2 1