Re: [Libguestfs] Use libguestfs with SAS URI of a disk image
On Sat, Mar 14, 2020 at 04:48:24AM +0000, JP Zhang wrote:
HI Richard,
Thanks for the quick reply.
The Python program that works, that I mentioned earlier, uses subprocess to issue/run "guestfish" commands below in that order. /libguestfs/run guestfish --listen -a <https storage-SAS-URL> -- ro then the Python code exported $PID to the environment. I believe so that the same $PID could be used for the subsequent guestfish calls. /libguestfs/run guestfish --remote -- -version /libguestfs/run guestfish --remote launch
Since it is running in the subprocess of "run.sh" shell, I couldn't see the libguestfs debug output. Maybe there is a way, I just don't know.
Then I manually typed similar guestfish commands in Linux bash shell, like guestfish --listen -a <https storage-SAS-URL> -- ro <this returned a $PID> guestfish --remote=<$PID> -- -version guestfish --remote=<$PID> launch this sequence of guestfish commands failed at "launch" cmd.
regarding params passed to add_drive call in libguestfs, the debug trace from running the guestfish cmds above looks like this: libguestfs: trace: add_drive "/<container>/abcd" "protocol:https" "server:tcp:<account>.blob.core.windows.net" which is something I already tried, not any more insights.
You'll need the trace from the working call, not the call that doesn't work. Try ‘export LIBGUESTFS_TRACE=1’ before the working command.
I really could not tell the difference between these two set of guestfish commands.
(1) do you see any substance difference between doing things in these two sets of commands? ones issued in Python program and ones issued in Bash shell manually? (2) if I can add a local VM disk image to libguestfs and list its files, can I safely say that I have libguestfs installed or set up correctly on my machine? meaning I should be able to do "network" based disk image calls/add "network" based disk image? correct, in theory? At this point, I cannot 100% trust my local linguestfs setup since I don't know much about libguestfs. (3) BTW, I see error message in manual run in Bash shell above. does this tell us anything? Could not access KVM kernel module: No such file or directory qemu-system-x86_64: failed to initialize KVM: No such file or directory qemu-system-x86_64: Back to tcg accelerator I see the same KVM error when I ran "libguestfs-test-tool" Is KVM kernel absolutely needed? or is this just an option to try and does libguestfs fall back to other Linux kernel to build supermin appliance
BTW, the working Python program was developed by others whom we have no contact now. So far I don't see anything special about Python. It is a just a shell program to issue libguestfs commands and wire up our own processing logic on top of libguestfs.
Regarding using "nbdkit", it looks like it is another service I need to run on my box. The goal of my project is to reduce the hops and improve the performance. Also the code I develop will be a web service that takes many real time requests. with nbdkit, does it mean that I need to create a instance of nbdkit on the fly for every request that my service receives? will nbdkit option work well when I have multiple concurrent libguestfs calls going through nbdkit in multiple threads/processes?
Actually nbdkit doesn't need to be run as a separate service. You can run it "captive" from the main program. It also has several advantages over using libguestfs to connect to curl directly. You probably want to look at what we do in virt-v2v, where we have almost exactly the same scenario and we use nbdkit to proxy the connection: https://github.com/libguestfs/virt-v2v/blob/master/v2v/nbdkit.ml https://github.com/libguestfs/virt-v2v/blob/f8e4e8643f105bc2abeb6b80b17a0ded... Also at: http://libguestfs.org/nbdkit.1.html http://libguestfs.org/nbdkit-captive.1.html http://libguestfs.org/nbdkit-curl-plugin.1.html http://libguestfs.org/nbdkit-cache-filter.1.html Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Hi Richard, Thank you! I really appreciate your help and information. J.P. ________________________________ From: Richard W.M. Jones <rjones@redhat.com> Sent: Saturday, March 14, 2020 3:04 AM To: JP Zhang <jpzhang@koono.com>; libguestfs@redhat.com <libguestfs@redhat.com> Subject: Re: Use libguestfs with SAS URI of a disk image On Sat, Mar 14, 2020 at 04:48:24AM +0000, JP Zhang wrote:
HI Richard,
Thanks for the quick reply.
The Python program that works, that I mentioned earlier, uses subprocess to issue/run "guestfish" commands below in that order. /libguestfs/run guestfish --listen -a <https storage-SAS-URL> -- ro then the Python code exported $PID to the environment. I believe so that the same $PID could be used for the subsequent guestfish calls. /libguestfs/run guestfish --remote -- -version /libguestfs/run guestfish --remote launch
Since it is running in the subprocess of "run.sh" shell, I couldn't see the libguestfs debug output. Maybe there is a way, I just don't know.
Then I manually typed similar guestfish commands in Linux bash shell, like guestfish --listen -a <https storage-SAS-URL> -- ro <this returned a $PID> guestfish --remote=<$PID> -- -version guestfish --remote=<$PID> launch this sequence of guestfish commands failed at "launch" cmd.
regarding params passed to add_drive call in libguestfs, the debug trace from running the guestfish cmds above looks like this: libguestfs: trace: add_drive "/<container>/abcd" "protocol:https" "server:tcp:<account>.blob.core.windows.net" which is something I already tried, not any more insights.
You'll need the trace from the working call, not the call that doesn't work. Try ‘export LIBGUESTFS_TRACE=1’ before the working command.
I really could not tell the difference between these two set of guestfish commands.
(1) do you see any substance difference between doing things in these two sets of commands? ones issued in Python program and ones issued in Bash shell manually? (2) if I can add a local VM disk image to libguestfs and list its files, can I safely say that I have libguestfs installed or set up correctly on my machine? meaning I should be able to do "network" based disk image calls/add "network" based disk image? correct, in theory? At this point, I cannot 100% trust my local linguestfs setup since I don't know much about libguestfs. (3) BTW, I see error message in manual run in Bash shell above. does this tell us anything? Could not access KVM kernel module: No such file or directory qemu-system-x86_64: failed to initialize KVM: No such file or directory qemu-system-x86_64: Back to tcg accelerator I see the same KVM error when I ran "libguestfs-test-tool" Is KVM kernel absolutely needed? or is this just an option to try and does libguestfs fall back to other Linux kernel to build supermin appliance
BTW, the working Python program was developed by others whom we have no contact now. So far I don't see anything special about Python. It is a just a shell program to issue libguestfs commands and wire up our own processing logic on top of libguestfs.
Regarding using "nbdkit", it looks like it is another service I need to run on my box. The goal of my project is to reduce the hops and improve the performance. Also the code I develop will be a web service that takes many real time requests. with nbdkit, does it mean that I need to create a instance of nbdkit on the fly for every request that my service receives? will nbdkit option work well when I have multiple concurrent libguestfs calls going through nbdkit in multiple threads/processes?
Actually nbdkit doesn't need to be run as a separate service. You can run it "captive" from the main program. It also has several advantages over using libguestfs to connect to curl directly. You probably want to look at what we do in virt-v2v, where we have almost exactly the same scenario and we use nbdkit to proxy the connection: https://github.com/libguestfs/virt-v2v/blob/master/v2v/nbdkit.ml https://github.com/libguestfs/virt-v2v/blob/f8e4e8643f105bc2abeb6b80b17a0ded... Also at: http://libguestfs.org/nbdkit.1.html http://libguestfs.org/nbdkit-captive.1.html http://libguestfs.org/nbdkit-curl-plugin.1.html http://libguestfs.org/nbdkit-cache-filter.1.html Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
participants (2)
-
JP Zhang -
Richard W.M. Jones