Re: [Libguestfs] Use libguestfs with SAS URI of a disk image
by Richard W.M. Jones
On Sat, Mar 14, 2020 at 04:48:24AM +0000, JP Zhang wrote:
> HI Richard,
>
> Thanks for the quick reply.
>
> The Python program that works, that I mentioned earlier, uses subprocess to issue/run "guestfish" commands below in that order.
> /libguestfs/run guestfish --listen -a <https storage-SAS-URL> -- ro
> then the Python code exported $PID to the environment. I believe so that the same $PID could be used for the subsequent guestfish calls.
> /libguestfs/run guestfish --remote -- -version
> /libguestfs/run guestfish --remote launch
>
> Since it is running in the subprocess of "run.sh" shell, I couldn't see the libguestfs debug output. Maybe there is a way, I just don't know.
>
> Then I manually typed similar guestfish commands in Linux bash shell, like
> guestfish --listen -a <https storage-SAS-URL> -- ro
> <this returned a $PID>
> guestfish --remote=<$PID> -- -version
> guestfish --remote=<$PID> launch
> this sequence of guestfish commands failed at "launch" cmd.
>
> regarding params passed to add_drive call in libguestfs, the debug trace from running the guestfish cmds above looks like this:
> libguestfs: trace: add_drive "/<container>/abcd" "protocol:https" "server:tcp:<account>.blob.core.windows.net"
> which is something I already tried, not any more insights.
You'll need the trace from the working call, not the
call that doesn't work.
Try ‘export LIBGUESTFS_TRACE=1’ before the working command.
> I really could not tell the difference between these two set of guestfish commands.
>
> (1) do you see any substance difference between doing things in these two sets of commands? ones issued in Python program and ones issued in Bash shell manually?
> (2) if I can add a local VM disk image to libguestfs and list its files, can I safely say that I have libguestfs installed or set up correctly on my machine? meaning I should be able to do "network" based disk image calls/add "network" based disk image? correct, in theory? At this point, I cannot 100% trust my local linguestfs setup since I don't know much about libguestfs.
> (3) BTW, I see error message in manual run in Bash shell above. does this tell us anything?
> Could not access KVM kernel module: No such file or directory
> qemu-system-x86_64: failed to initialize KVM: No such file or directory
> qemu-system-x86_64: Back to tcg accelerator
> I see the same KVM error when I ran "libguestfs-test-tool"
> Is KVM kernel absolutely needed? or is this just an option to try and does libguestfs fall back to other Linux kernel to build supermin appliance
>
> BTW, the working Python program was developed by others whom we have no contact now. So far I don't see anything special about Python. It is a just a shell program to issue libguestfs commands and wire up our own processing logic on top of libguestfs.
>
> Regarding using "nbdkit", it looks like it is another service I need to run on my box. The goal of my project is to reduce the hops and improve the performance. Also the code I develop will be a web service that takes many real time requests. with nbdkit, does it mean that I need to create a instance of nbdkit on the fly for every request that my service receives? will nbdkit option work well when I have multiple concurrent libguestfs calls going through nbdkit in multiple threads/processes?
Actually nbdkit doesn't need to be run as a separate service. You can
run it "captive" from the main program. It also has several
advantages over using libguestfs to connect to curl directly.
You probably want to look at what we do in virt-v2v, where we have
almost exactly the same scenario and we use nbdkit to proxy the
connection:
https://github.com/libguestfs/virt-v2v/blob/master/v2v/nbdkit.ml
https://github.com/libguestfs/virt-v2v/blob/f8e4e8643f105bc2abeb6b80b17a0...
Also at:
http://libguestfs.org/nbdkit.1.html
http://libguestfs.org/nbdkit-captive.1.html
http://libguestfs.org/nbdkit-curl-plugin.1.html
http://libguestfs.org/nbdkit-cache-filter.1.html
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
4 years, 9 months
[PATCH v3 00/34] Configurable policy for handling deprecated interfaces
by Markus Armbruster
This series extends QMP introspection to cover deprecation.
Additionally, new option -compat lets you configure what to do when
deprecated interfaces get used. This is intended for testing users of
the management interfaces. It is experimental.
-compat deprecated-input=<in-policy> configures what to do when
deprecated input is received. Available policies:
* accept: Accept deprecated commands and arguments (default)
* reject: Reject them
* crash: Crash
-compat deprecated-output=<out-policy> configures what to do when
deprecated output is sent. Available output policies:
* accept: Emit deprecated command results and events (default)
* hide: Suppress them
For now, -compat covers only deprecated syntactic aspects of QMP. We
may want to extend it to cover semantic aspects, CLI, and experimental
features.
PATCH 01-04: Documentation fixes
PATCH 05-10: Test improvements
PATCH 11-24: Add feature flags to remaining user-defined types and to
struct members
PATCH 25-26: New special feature 'deprecated', visible in
introspection
PATCH 27-34: New -compat to set policy for handling stuff marked with
feature 'deprecated'
v3:
* Rebased, non-trivial conflicts in PATCH 01+26+27+34 due to RST
conversion and code motion
* PATCH 28-29: Old PATCH 28 split up to ease review
* PATCH 30-31: New
* PATCH 32-33: Old PATCH 29 split up to ease review
Comparison to RFC (24 Oct 2019):
* Cover arguments and results in addition to commands and events
* Half-baked "[RFC PATCH 18/19] qapi: Include a warning in the
response to a deprecated command" dropped
See also last item of
Subject: Minutes of KVM Forum BoF on deprecating stuff
Date: Fri, 26 Oct 2018 16:03:51 +0200
Message-ID: <87mur0ls8o.fsf(a)dusky.pond.sub.org>
https://lists.nongnu.org/archive/html/qemu-devel/2018-10/msg05828.html
Cc: Lukáš Doktor <ldoktor(a)redhat.com>
Cc: libguestfs(a)redhat.com
Cc: libvir-list(a)redhat.com
Cc: Daniel P. Berrange <berrange(a)redhat.com>
Cc: Peter Krempa <pkrempa(a)redhat.com>
Markus Armbruster (34):
qemu-doc: Belatedly document QMP command arg & result deprecation
qapi: Belatedly update doc comment for @wait deprecation
docs/devel/qapi-code-gen: Clarify allow-oob introspection
docs/devel/qapi-code-gen: Document 'features' introspection
tests/test-qmp-cmds: Factor out qmp_dispatch() test helpers
tests/test-qmp-cmds: Check responses more thoroughly
tests/test-qmp-cmds: Simplify test data setup
tests/test-qmp-event: Simplify test data setup
tests/test-qmp-event: Use qobject_is_equal()
tests/test-qmp-event: Check event is actually emitted
qapi/schema: Clean up around QAPISchemaEntity.connect_doc()
qapi: Add feature flags to remaining definitions
qapi: Consistently put @features parameter right after @ifcond
qapi/introspect: Rename *qlit* to reduce confusion
qapi/introspect: Factor out _make_tree()
qapi/schema: Change _make_features() to a take feature list
qapi/schema: Reorder classes so related ones are together
qapi/schema: Rename QAPISchemaObjectType{Variant,Variants}
qapi/schema: Call QAPIDoc.connect_member() in just one place
qapi: Add feature flags to struct members
qapi: Inline do_qmp_dispatch() into qmp_dispatch()
qapi: Simplify how qmp_dispatch() deals with QCO_NO_SUCCESS_RESP
qapi: Simplify how qmp_dispatch() gets the request ID
qapi: Replace qmp_dispatch()'s TODO comment by an explanation
qapi: New special feature flag "deprecated"
qapi: Mark deprecated QMP parts with feature 'deprecated'
qemu-options: New -compat to set policy for deprecated interfaces
qapi: Implement deprecated-output=hide for QMP command results
qapi: Implement deprecated-output=hide for QMP events
qapi: Implement deprecated-output=hide for QMP event data
qapi: Implement deprecated-output=hide for QMP introspection
qapi: Implement deprecated-input=reject for QMP commands
qapi: Implement deprecated-input=reject for QMP command arguments
qapi: New -compat deprecated-input=crash
docs/devel/qapi-code-gen.txt | 79 ++-
docs/system/deprecated.rst | 48 +-
tests/qapi-schema/doc-good.texi | 32 ++
qapi/block-core.json | 48 +-
qapi/block.json | 30 +-
qapi/char.json | 1 +
qapi/compat.json | 52 ++
qapi/control.json | 11 +-
qapi/introspect.json | 28 +-
qapi/machine.json | 34 +-
qapi/migration.json | 36 +-
qapi/misc.json | 13 +-
qapi/qapi-schema.json | 1 +
include/qapi/compat-policy.h | 20 +
include/qapi/qmp/dispatch.h | 1 +
include/qapi/qobject-input-visitor.h | 9 +
include/qapi/qobject-output-visitor.h | 9 +
include/qapi/visitor-impl.h | 3 +
include/qapi/visitor.h | 9 +
monitor/monitor-internal.h | 3 -
monitor/misc.c | 2 -
monitor/qmp-cmds-control.c | 102 +++-
qapi/qapi-visit-core.c | 9 +
qapi/qmp-dispatch.c | 137 ++---
qapi/qobject-input-visitor.c | 29 ++
qapi/qobject-output-visitor.c | 20 +
qemu-storage-daemon.c | 2 -
softmmu/vl.c | 17 +
tests/test-qmp-cmds.c | 249 +++++----
tests/test-qmp-event.c | 201 +++-----
qapi/Makefile.objs | 8 +-
qapi/trace-events | 1 +
qemu-options.hx | 22 +
scripts/qapi/commands.py | 20 +-
scripts/qapi/doc.py | 16 +-
scripts/qapi/events.py | 22 +-
scripts/qapi/expr.py | 14 +-
scripts/qapi/introspect.py | 104 ++--
scripts/qapi/schema.py | 488 ++++++++++--------
scripts/qapi/types.py | 8 +-
scripts/qapi/visit.py | 28 +-
tests/Makefile.include | 1 +
tests/qapi-schema/alternate-base.err | 2 +-
tests/qapi-schema/doc-good.json | 22 +-
tests/qapi-schema/doc-good.out | 18 +
.../qapi-schema/features-deprecated-type.err | 2 +
.../qapi-schema/features-deprecated-type.json | 3 +
.../qapi-schema/features-deprecated-type.out | 0
tests/qapi-schema/qapi-schema-test.json | 51 +-
tests/qapi-schema/qapi-schema-test.out | 48 +-
tests/qapi-schema/test-qapi.py | 26 +-
51 files changed, 1384 insertions(+), 755 deletions(-)
create mode 100644 qapi/compat.json
create mode 100644 include/qapi/compat-policy.h
create mode 100644 tests/qapi-schema/features-deprecated-type.err
create mode 100644 tests/qapi-schema/features-deprecated-type.json
create mode 100644 tests/qapi-schema/features-deprecated-type.out
--
2.21.1
4 years, 9 months
Re: [Libguestfs] Use libguestfs with SAS URI of a disk image
by Richard W.M. Jones
On Fri, Mar 13, 2020 at 09:41:37PM +0000, JP Zhang wrote:
> Hi Richard,
>
> Thanks for the great work you did on libguestfs.
>
> I try to call libguestfs lib API directly in C to "add drive" by
> using a Azure blob storage access SAS URI, which has shared access
> token/secret on the URL.
>
> The URL likes like this:
> 'https://<account>.blob.core.windows.net/<container>/<VMDiskImage>?sv=2017-04-17&sr=b&si=<guid>&sig=<secret>'
> I try to use this API:
>
> int
> guestfs_add_drive_opts (guestfs_h *g,
> const char *filename,
> ...);
>
>
> My code looks like this:
> char const *servers[] = { (char const *)"<account>.blob.core.windows.net", NULL };
> int addDriveRet = guestfs_add_drive_opts (g, "/<container>/<VMDiskImage>?sv=2017-04-17&sr=b&si=<guid>&sig=<secret>",
> GUESTFS_ADD_DRIVE_OPTS_FORMAT, "raw",
> GUESTFS_ADD_DRIVE_OPTS_PROTOCOL, "https",
> GUESTFS_ADD_DRIVE_OPTS_SERVER, servers,
> GUESTFS_ADD_DRIVE_OPTS_READONLY, 1,
> -1);
Since you say you got it to work with guestfish, one trick is to run
‘guestfish -x’ (or the ‘trace true’ command, or set LIBGUESTFS_TRACE=1)
and it will actually print the full API call you should use.
I believe the problem may be to do with URI %-quoting of the filename
(2nd) parameter.
> I read the documentation and samples on libguestfs.org in depth. I
> could not find a good sample for "https" with SAS secret on URL. I
> tried many different ways to break up the URL into "server",
> "protocol", and secret. It looks like "secret" option is only for
> "rbd". From the debug log/printouts from libguestfs, it seemed to
> reassemble the parts of original URL back to a URL that matched the
> original one. But I got http 403 error( auth error). I am very sure
> the URL is correct because I can download the disk image using
> Curl. The libguestfs debug log printout displayed the URL with ?
> being turned into %3F and % into %25. I am not sure whether
> libguestfs URL-encoded printout was just for printout or whether it
> was the actual URL string being used to make the https server
> call. If it was, then the secret has been changed due to URL
> encoding again. This is just a guess. Maybe not related at all.
>
> BTW, I can run libguestfs-test-tool successfully. and I have another
> program in Python that call through /run.sh -> GuestFish ->
> libguestfs lib, it works with the same original SAS URL.
So you can contact the SAS URL through this? If so I would try the
trace trick above to see the actual protocol call being made.
If that doesn't work, another way to do it would be to
forward the connection through nbdkit, ie:
rm -f /tmp/socket
nbdkit -f -U /tmp/socket curl 'https://...' &
and connect to the the NBD socket using the add_drive_opts API as
documented in the manual.
(in guestfish you'd use: guestfish -a 'nbd://?socket=/tmp/socket')
nbdkit can also be run captive from your program, see
http://libguestfs.org/nbdkit-captive.1.html
> also, on my dev machine, I can do "guestfs_add_drive" on the local
> disk image successfully and mount the device, and list files after I
> downloaded the same disk image to my local machine.
>
> guestfs_add_drive(g, "/pathto/vmdiskimage");
>
> Could you share how I should fill in various params in
> "guest_add_drive_opts()? like "PROTOCOL", "SERVER", "filename"( the
> 2nd param), or USERNAME ( not seem to apply here, I also tried with
> putting "secret" in SECRET, "sig" in USERNAME, the rebuilt URL from
> libguestfs looks missing these parts in final URL, it probably put
> these in HTTP header.)
>
> I also try to call guestfs_add_drive() with the original URL
> below. it didn't work either.
>
> guestfs_add_drive(g, "https://<account>.blob.core.windows.net/<container>/<VMDiskImage>?sv=2017-04-17&sr=b&si=<guid>&sig=<secret>");
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
4 years, 9 months
[libnbd PATCH] lib: remove extra @LIBS@ from pkg-config file
by Pino Toscano
At the moment it is empty, so probably it does not exist. Remove it to
avoid adding spurious content to the pkg-config file in case that
variable will get a value in the future.
---
lib/libnbd.pc.in | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/libnbd.pc.in b/lib/libnbd.pc.in
index ba91f49..294c090 100644
--- a/lib/libnbd.pc.in
+++ b/lib/libnbd.pc.in
@@ -8,4 +8,4 @@ Version: @PACKAGE_VERSION@
Description: NBD client library in userspace
Requires:
Cflags:
-Libs: -lnbd @LIBS@
+Libs: -lnbd
--
2.24.1
4 years, 9 months
[PATCH] get-kernel: fallback to uncompressed vmlinux kernels
by Pino Toscano
In case the Linux guest has no compressed linux kernel (vmlinuz),
fallback to uncompressed ones (vmlinux).
Reported by: Laurent Vivier.
---
get-kernel/get_kernel.ml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/get-kernel/get_kernel.ml b/get-kernel/get_kernel.ml
index 3015c0fca..e485cf495 100644
--- a/get-kernel/get_kernel.ml
+++ b/get-kernel/get_kernel.ml
@@ -144,6 +144,9 @@ and pick_kernel_files_linux (g : Guestfs.guestfs) root =
let kernels = glob "/boot/vmlinuz-*" in
let initrds = glob "/boot/initramfs-*" in
+ (* Uncompressed kernels: *)
+ let kernels = if kernels <> [] then kernels else glob "/boot/vmlinux-*" in
+
(* Old RHEL: *)
let initrds = if initrds <> [] then initrds else glob "/boot/initrd-*" in
--
2.24.1
4 years, 9 months
[hivex PATCH] Remove extra @LIBS@ from pkg-config file
by Pino Toscano
At the moment it is empty, so probably it does not exist. Remove it to
avoid adding spurious content to the pkg-config file in case that
variable will get a value in the future.
---
hivex.pc.in | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hivex.pc.in b/hivex.pc.in
index c5f594a..250a58b 100644
--- a/hivex.pc.in
+++ b/hivex.pc.in
@@ -8,4 +8,4 @@ Version: @VERSION@
Description: Read and write Windows Registry Hive files.
Requires:
Cflags:
-Libs: -lhivex @LIBS@
+Libs: -lhivex
--
2.24.1
4 years, 9 months
[PATCH virt-v2v] v2v: Use LIBGUESTFS_CFLAGS/LIBS when compiling and linking.
by Richard W.M. Jones
Allows virt-v2v to be compiled against the libguestfs build directory.
---
v2v/Makefile.am | 2 ++
1 file changed, 2 insertions(+)
diff --git a/v2v/Makefile.am b/v2v/Makefile.am
index 2a196d45d..104420d13 100644
--- a/v2v/Makefile.am
+++ b/v2v/Makefile.am
@@ -197,6 +197,7 @@ virt_v2v_CPPFLAGS = \
-I$(top_srcdir)/lib
virt_v2v_CFLAGS = \
$(WARN_CFLAGS) $(WERROR_CFLAGS) \
+ $(LIBGUESTFS_CFLAGS) \
$(LIBVIRT_CFLAGS) \
$(LIBOSINFO_CFLAGS)
@@ -230,6 +231,7 @@ endif
OCAMLCLIBS = \
-lqemuopts \
+ $(LIBGUESTFS_LIBS) \
$(LIBVIRT_LIBS) \
$(LIBXML2_LIBS) \
$(JANSSON_LIBS) \
--
2.24.1
4 years, 9 months
[PATCH] lib: remove extra @LIBS@ from pkg-config file
by Pino Toscano
At the moment it is empty, so probably it does not exist. Remove it to
avoid adding spurious content to the pkg-config file in case that
variable will get a value in the future.
---
lib/libguestfs.pc.in | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/libguestfs.pc.in b/lib/libguestfs.pc.in
index 679ed7eba..013bf0f28 100644
--- a/lib/libguestfs.pc.in
+++ b/lib/libguestfs.pc.in
@@ -8,4 +8,4 @@ Version: @VERSION@
Description: libguestfs library for accessing and modifying VM images
Requires:
Cflags:
-Libs: -lguestfs @LIBS@
+Libs: -lguestfs
--
2.24.1
4 years, 9 months
[PATCH] builder: templates: add the AppStream repo
by Pino Toscano
It is generally used together with BaseOS (which contains a restricted
set of packages), so add it to the preconfigured repositories by
default.
---
builder/templates/make-template.ml | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/builder/templates/make-template.ml b/builder/templates/make-template.ml
index 8d26177bf..a4e2f37fb 100755
--- a/builder/templates/make-template.ml
+++ b/builder/templates/make-template.ml
@@ -1238,7 +1238,8 @@ and make_rhel_yum_conf major minor arch =
major major minor in
sprintf "%s/Server/%s/os" topurl arch,
sprintf "%s/source/SRPMS" topurl,
- Some (sprintf "%s/Server/optional/%s/os" arch topurl,
+ Some ("Optional",
+ sprintf "%s/Server/optional/%s/os" arch topurl,
sprintf "%s/Server/optional/source/SRPMS" topurl)
| 7, (X86_64|PPC64|PPC64le|S390X) ->
let topurl =
@@ -1246,7 +1247,8 @@ and make_rhel_yum_conf major minor arch =
major major minor in
sprintf "%s/Server/%s/os" topurl (string_of_arch arch),
sprintf "%s/Server/source/tree" topurl,
- Some (sprintf "%s/Server-optional/%s/os" topurl (string_of_arch arch),
+ Some ("Optional",
+ sprintf "%s/Server-optional/%s/os" topurl (string_of_arch arch),
sprintf "%s/Server-optional/source/tree" topurl)
| 7, Aarch64 ->
let topurl =
@@ -1254,7 +1256,8 @@ and make_rhel_yum_conf major minor arch =
major major minor in
sprintf "%s/Server/%s/os" topurl (string_of_arch arch),
sprintf "%s/Server/source/tree" topurl,
- Some (sprintf "%s/Server-optional/%s/os" topurl (string_of_arch arch),
+ Some ("Optional",
+ sprintf "%s/Server-optional/%s/os" topurl (string_of_arch arch),
sprintf "%s/Server-optional/source/tree" topurl)
| 8, arch ->
let topurl =
@@ -1262,7 +1265,9 @@ and make_rhel_yum_conf major minor arch =
major major minor in
sprintf "%s/BaseOS/%s/os" topurl (string_of_arch arch),
sprintf "%s/BaseOS/source/tree" topurl,
- None (* XXX sort out AppStream and CRB *)
+ Some ("AppStream",
+ sprintf "%s/AppStream/%s/os" topurl (string_of_arch arch),
+ sprintf "%s/AppStream/source/tree" topurl)
| _ -> assert false in
bpf "\
@@ -1285,23 +1290,25 @@ keepcache=0
(match optional with
| None -> ()
- | Some (optionalbaseurl, optionalsrpms) ->
+ | Some (name, optionalbaseurl, optionalsrpms) ->
+ let lc_name = String.lowercase_ascii name in
bpf "\
-[rhel%d-optional]
-name=RHEL %d Server Optional
+[rhel%d-%s]
+name=RHEL %d Server %s
baseurl=%s
enabled=1
gpgcheck=0
keepcache=0
-[rhel%d-optional-source]
-name=RHEL %d Server Optional
+[rhel%d-%s-source]
+name=RHEL %d Server %s
baseurl=%s
enabled=0
gpgcheck=0
keepcache=0
-" major major optionalbaseurl major major optionalsrpms
+" major lc_name major lc_name optionalbaseurl
+ major lc_name major lc_name optionalsrpms
)
) else (
assert false (* not implemented for RHEL major >= 9 *)
--
2.24.1
4 years, 9 months
