[PATCH v2 00/11] rvh-upload: Various fixes and cleanups
by Nir Soffer
This series extract oVirt SDK and imageio code to make it eaiser to follow the
code and improve error handing in open() and close().
Tested with virt-v2v master.
Changes since v1:
- Rebase on merged patches from v1
- Fix regression introduced by "rhv-upload: Fix cleanup after errors"
- Remove "rhv-upload: Try to remove disk on timeout" since it cannot
succeed
- Add more context in debug error logs
- Use disk_id instead of disk.id in close()
v1 was here:
https://www.redhat.com/archives/libguestfs/2019-November/msg00060.html
Nir Soffer (11):
rhv-upload: Cancel transfer if finalize failed
rhv-upload: Keep disk_id in handle
rhv-upload: Show disk id in error message
rhv-upload: Don't keep transfer_service in handle
rhv-upload: Get host before creating disk
rhv-upload: Extract create_transfer() function
rhv-upload: Show transfer id in error message
rhv-upload: Extract imageio helpers
rhv-upload: Extract get_options() helper
rhv-upload: Extract optimize_http() helper
rhv-upload: Clean up username and password
v2v/rhv-upload-plugin.py | 325 ++++++++++++++++++++++-----------------
1 file changed, 184 insertions(+), 141 deletions(-)
--
2.21.0
4 years, 6 months
Building libguestfs from git
by Nir Soffer
Looks like libguestfs build is broken, or not documented properly.
I tried (based on http://libguestfs.org/guestfs-building.1.html):
git clean -dxf
./autogen.sh
autoreconf failes with:
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:38: installing 'build-aux/missing'
configure.ac:221: error: required file
'common/mlcustomize/Makefile.in' not found
configure.ac:221: error: required file 'common/mllibvirt/Makefile.in' not found
configure.ac:221: error: required file
'common/mlstdutils/guestfs_config.ml.in' not found
configure.ac:221: error: required file 'common/mlv2v/Makefile.in' not found
Makefile.am:161: error: required directory ./common/mlcustomize does not exist
Makefile.am:161: error: required directory ./common/mlv2v does not exist
Makefile.am:169: error: required directory ./common/mllibvirt does not exist
parallel-tests: installing 'build-aux/test-driver'
align/Makefile.am: installing 'build-aux/depcomp'
configure.ac: installing 'build-aux/ylwrap'
autoreconf: automake failed with exit status: 1
But the autogen.sh continue, ignoring the failure...
Finally it fails with:
config.status: error: cannot find input file: `Makefile.in'
I tried to run it again:
./autogen.sh
the autoreconf errors show again, but this time it succeeds, but:
$ make -j8
cfg.mk:139: *** gnulib update required; run ./autogen.sh first. Stop.
Running ./autoge.sh again does not help.
I can build tag v1.40.2, but it is missing some patches in virt-v2v that I need.
Thanks,
Nir
4 years, 6 months
How to build virt-v2v after the project was separated
by Nir Soffer
Looks like virt-v2v cannot be built now, since it requires non-existing version
of libguestfs-devel.
Based on (no README or any instructions in virt-v2v)
http://libguestfs.org/guestfs-building.1.html
I tried this:
git clean -dxf
./autogen.sh
And it fails with:
--- Checking for libraries used by virt-v2v ---
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for LIBGUESTFS... no
configure: error: Package requirements (libguestfs >= 1.41.5) were not met:
Package dependency requirement 'libguestfs >= 1.41.5' could not be satisfied.
Package 'libguestfs' has version '1.40.2', required version is '>= 1.41.5'
$ rpm -qa | grep libguestfs
libguestfs-xfs-1.40.2-4.fc30.x86_64
libguestfs-tools-c-1.40.2-4.fc30.x86_64
libguestfs-devel-1.40.2-4.fc30.x86_64
libguestfs-1.40.2-4.fc30.x86_64
Is it possible to build using libguestfs source?
Nir
4 years, 6 months
[PATCH 00/18] rvh-upload: Various fixes and cleanups
by Nir Soffer
This series extract oVirt SDK and imageio code to make it eaiser to follow the
code and improve error handing in open() and close().
The first small patches can be consider as fixes for downstream.
Tested based on libguestfs v1.41.5, since I had trouble building virt-v2v and
libguestfs from master.
Nir Soffer (18):
rhv-upload: Remove unused exception class
rhv-upload: Check status more frequently
rhv-upload: Don't flush() in close()
rhv-upload: Ensure connection is closed in close()
rhv-upload: Ensure http is closed in close()
rhv-upload: Fix cleanup after errors
rhv-upload: Group oVirt SDK functions
rhv-upload: Don't keep disk_service in handle
rhv-upload: Extract create_disk() function
rhv-upload: Try to remove disk on timeout
rhv-upload: Keep disk_id in handle
rhv-upload: Don't keep transfer_service in handle
rhv-upload: Get host before creating disk
rhv-upload: Extract create_transfer() function
rhv-upload: Extract imageio helpers
rhv-upload: Extract get_options() helper
rhv-upload: Extract optimize_http() helper
rhv-upload: Clean up username and password
v2v/rhv-upload-plugin.py | 550 +++++++++++++++++++++------------------
1 file changed, 295 insertions(+), 255 deletions(-)
--
2.21.0
4 years, 6 months
[PATCH 1/2] options: Fixes and enhancements to --key parsing.
by Richard W.M. Jones
The first patch fixes a rather serious bug, the second patch allows
multiple --key parameters and default parameters.
There is a third patch to libguestfs which adds a test, coming up.
I did not yet review and fix the documentation. I think we need to
centralize it in one place because at the moment the same
documentation for --key is copy/pasted all over the tools.
Rich.
4 years, 6 months
[PATCH] Python: Fix GIL usage in guestfs_int_py_event_callback_wrapper (RHBZ#1773520)
by Sam Eiderman
All Py_* functions should be protected by the GIL.
Otherwise internal python data structures can get corrupted.
Move PyGILState_Ensure to the beginning of the block and
PyGILState_Release to the bottom.
Signed-off-by: Sam Eiderman <sameid(a)google.com>
---
python/handle.c | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/python/handle.c b/python/handle.c
index ffd0794d8..659bb5b06 100644
--- a/python/handle.c
+++ b/python/handle.c
@@ -112,13 +112,17 @@ guestfs_int_py_event_callback_wrapper (guestfs_h *g,
const char *buf, size_t buf_len,
const uint64_t *array, size_t array_len)
{
- PyGILState_STATE py_save = PyGILState_UNLOCKED;
+ PyGILState_STATE py_save;
PyObject *py_callback = callback;
PyObject *py_array;
PyObject *args;
PyObject *a;
size_t i;
PyObject *py_r;
+ int threads_initialized = PyEval_ThreadsInitialized ();
+
+ if (threads_initialized)
+ py_save = PyGILState_Ensure ();
py_array = PyList_New (array_len);
for (i = 0; i < array_len; ++i) {
@@ -132,14 +136,8 @@ guestfs_int_py_event_callback_wrapper (guestfs_h *g,
buf, buf_len, py_array);
Py_INCREF (args);
- if (PyEval_ThreadsInitialized ())
- py_save = PyGILState_Ensure ();
-
py_r = PyObject_CallObject (py_callback, args);
- if (PyEval_ThreadsInitialized ())
- PyGILState_Release (py_save);
-
Py_DECREF (args);
if (py_r != NULL)
@@ -147,6 +145,9 @@ guestfs_int_py_event_callback_wrapper (guestfs_h *g,
else
/* Callback threw an exception: print it. */
PyErr_PrintEx (0);
+
+ if (threads_initialized)
+ PyGILState_Release (py_save);
}
PyObject *
--
2.24.0.432.g9d3f5f5b63-goog
4 years, 6 months
ANNOUNCE: libnbd 1.2 & nbdkit 1.16 - high performance NBD client and server
by Richard W.M. Jones
I'm pleased to announce the releases of libnbd 1.2 and nbdkit 1.16.
These are a high performance Network Block Device (NBD) client library
and server.
Key features of libnbd:
* Synchronous API for ease of use.
* Asynchronous API for writing non-blocking, multithreaded clients.
You can mix both APIs freely.
* High performance.
* Minimal dependencies for the basic library.
* Well-documented, stable API.
* Bindings in several programming languages.
* Shell (nbdsh) for command line and scripting.
Git: https://github.com/libguestfs/libnbd
Download: http://download.libguestfs.org/libnbd/1.2-stable/
Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=28807
Key features of nbdkit:
* Multithreaded NBD server written in C with good performance.
* Minimal dependencies for the basic server.
* Liberal license (BSD) allows nbdkit to be linked to proprietary
libraries or included in proprietary code.
* Well-documented, simple plugin API with a stable ABI guarantee.
Lets you export “unconventional” block devices easily.
* You can write plugins in C, Lua, Perl, Python, OCaml, Ruby, Rust,
shell script or Tcl.
* Filters can be stacked in front of plugins to transform the output.
Git: https://github.com/libguestfs/nbdkit
Download: http://download.libguestfs.org/nbdkit/1.16-stable/
Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=16469
*** Release notes for libnbd 1.2 ***
These are the release notes for libnbd stable release 1.2. This
describes the major changes since 1.0.
libnbd 1.2.0 was released on 14th November 2019.
Security
Two security problems were found during development of libnbd 1.2.
Both were backported to the 1.0 stable branch. Upgrading is highly
advisable.
CVE-2019-14842 protocol downgrade attack when using
"LIBNBD_TLS_REQUIRE"
See the full announcement and links to mitigation, tests and fixes
here:
https://www.redhat.com/archives/libguestfs/2019-September/msg00128.html
remote code execution vulnerability
See the full announcement here:
https://www.redhat.com/archives/libguestfs/2019-October/msg00060.html
New APIs
nbd_can_fast_zero(3)
Test support by the server for fast zeroing (Eric Blake).
nbd_connect_socket(3)
nbd_aio_connect_socket(3)
Connect to a local connected socket which you create in your main
program using your own chosen method.
nbd_connect_systemd_socket_activation(3)
nbd_aio_connect_systemd_socket_activation(3)
Connect to local processes that support systemd socket activation.
nbd_connect_vsock(3)
nbd_aio_connect_vsock(3)
Used to connect to servers over "AF_VSOCK".
nbd_get_handshake_flags(3)
nbd_set_handshake_flags(3)
nbd_get_request_structured_replies(3)
nbd_set_request_structured_replies(3)
nbd_get_structured_replies_negotiated(3)
Can be used when testing NBD servers to avoid various NBD features
(Eric Blake).
nbd_get_protocol(3)
Get the NBD protocol variant that the server supports.
nbd_get_tls_negotiated(3)
Did we actually negotiate a TLS connection?
nbd_set_uri_allow_local_file(3)
nbd_set_uri_allow_tls(3)
nbd_set_uri_allow_transports(3)
These can be used to filter NBD URIs before calling
nbd_connect_uri(3).
New features
New tool nbdfuse(1) lets you create a loop-mounted file backed by an
NBD server without needing root.
"AF_VSOCK" is now a supported protocol (thanks Stefan Hajnoczi and
Stefano Garzarella).
Support for the "FAST_ZERO" flag (Eric Blake).
Allow disabling certain protocol features, to make it easier to test
servers (Eric Blake).
Stack-allocated Variable Length Arrays (VLAs) are now banned throughout
the library, making the library easier to consume from threads and
other small stack situations.
Reproducible builds (Chris Lamb).
Support for filtering potentially dangerous or undesirable NBD URI
features.
Documentation
Many improvements to the generated manual pages, including:
· Separate "RETURN VALUE" and "ERRORS" sections for each API
function.
· Example code.
· Relevant links can be added to the "SEE ALSO" section.
· Link to NBD URI specification where relevant, and improve
documentation around what URIs libnbd supports.
· Document libnbd version number scheme.
· Document limits on export name length, encoding etc.
New libnbd-security(3) man page listing past security issues and
remediations (Eric Blake).
Tools
nbdsh(1) has a new --base-allocation option which can be used to
request "base:allocation" metadata context.
New nbdsh(1) --uri (-u) option to connect to URIs.
Tests
You can now fuzz libnbd using either American Fuzzy Lop or clang’s
libFuzzer.
Add unit tests for nbdsh(1) (Eric Blake).
Improved interop testing with various NBD servers and features.
Other improvements and bug fixes
nbd_connect_tcp(3) now tries to return the correct errno(3) from the
underlying connect(2) call when that fails.
The nbd-protocol.h header file is now shared between libnbd and nbdkit.
Better fork-safety in "nbd_connect_*" APIs.
The code was analyzed with Coverity and various problems identified and
fixed.
*** Release notes for nbdkit 1.16 ***
These are the release notes for nbdkit stable release 1.16. This
describes the major changes since 1.14.
nbdkit 1.16.0 was released on 14th November 2019.
Security
Two security issues were found during development of nbdkit 1.16.
Fixes for these were backported to older stable branches. Upgrading to
the fixed versions is highly recommended. The new nbdkit-security(1)
man page contains an up to date list of past security issues.
CVE-2019-14850 denial of service due to premature opening of back-end
connection
See the full announcement and links to mitigation, tests and fixes
here:
https://www.redhat.com/archives/libguestfs/2019-September/msg00084.html
CVE-2019-14851 assertion failure by issuing commands in the wrong order
This CVE was caused by the fix to the previous issue.
See the full announcement and links to mitigation, tests and fixes
here:
https://www.redhat.com/archives/libguestfs/2019-September/msg00272.html
New features
Add support for fast zeroing. Plugins can expose this using the new
".can_fast_zero" method (Eric Blake).
nbdkit-partitioning-plugin(1) allows use of "mbr-id=default" or
"type-guid=default" to go back to the default MBR byte or partition
type GUID.
New --mask-handshake server flag can be used for testing client feature
negotiation (Eric Blake).
The client export name is passed to nbdkit-captive(1) --run parameter
as $exportname (Eric Blake).
Captive --run commands which fail (eg. aborting) now cause nbdkit to
exit with an error instead of errors being silently ignored (Eric
Blake).
File descriptors can be passed to password parameters, eg:
"password=-3" which means that the password should be read from file
descriptor 3.
nbdkit can now serve over the "AF_VSOCK" protocol (thanks Stefan
Hajnoczi).
New --log=null option discards error messages.
Plugins
Python 2 support has been dropped from nbdkit-python-plugin(3) in line
with Python 2 end of life at the beginning of 2020. Python ≥ 3.3 is
required by this plugin. If you wish to continue to use Python 2 then
you will need to use nbdkit 1.14.
New nbdkit-info-plugin(1) which returns various server information back
to the client. It can be used for testing server latency amongst other
things.
nbdkit-data-plugin(1) now allows you to write "BYTE*N" to get repeated
bytes (eg. nbdkit data data="0x55*4096").
nbdkit-ssh-plugin(1) new parameter "compression=true|false" to control
transport compression.
nbdkit-vddk-plugin(1) is no longer compiled on non-x86 platforms since
VMware has only ever shipped VDDK on x86.
nbdkit-sh-plugin(1) scripts can now see the client exportname and can
use the "magic_config_key" feature.
Filters
New nbdkit-retry-filter(1) which can reopen the plugin transparently on
certain types of failures (lots of help from Eric Blake).
API
Macros "NBDKIT_VERSION_MAJOR", "NBDKIT_VERSION_MINOR",
"NBDKIT_VERSION_MICRO" expose the compile-time version of nbdkit to
plugins and filters (Eric Blake).
Filters (which unlike plugins do not have a public stable API) must now
exactly match the version of nbdkit when loaded (Eric Blake).
New ".can_fast_zero" method (Eric Blake).
New "nbdkit_export_name" server function for reading the export name
passed by the client.
New "nbdkit_peer_name" server function to return the client address
(like getpeername(2)).
New server functions for safely parsing integers: "nbdkit_parse_int",
"nbdkit_parse_unsigned", "nbdkit_parse_int8_t", "nbdkit_parse_uint8_t",
"nbdkit_parse_int16_t", "nbdkit_parse_uint16_t",
"nbdkit_parse_int32_t", "nbdkit_parse_uint32_t",
"nbdkit_parse_int64_t", "nbdkit_parse_uint64_t".
Bug fixes
".trim" with FUA flag set now works (Eric Blake).
Documentation
The previous release notes have been turned into man pages.
Tests
Several tests now optionally use nbdsh(1) instead of qemu-io.
You can now fuzz nbdkit using either American Fuzzy Lop or clang’s
libFuzzer.
Several tests have had sleep times increased to make them more stable
when run on slow or heavily loaded machines.
Internals
Reproducible builds (Chris Lamb).
Compile code with -Wshadow warning (Eric Blake).
The internal backend system has been extensively overhauled. In
particular this means that we now validate request ranges as requests
are passed between filters and down to the plugin, making it easier to
find bugs in filters early (Eric Blake).
Plugin size and "can_*" flags are cached more aggressively by the
server (Eric Blake).
Variable Length Arrays (VLAs) on stack are now banned throughout the
code.
The nbd-protocol.h header describing the NBD protocol is now shared
with libnbd(3).
Plugin ".unload" method is now called after all worker threads have
exited, avoiding races at server shutdown.
Code was audited using Coverity and various problems were fixed.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
4 years, 6 months
[PATCH v2v v2] nbdkit: Use cacheextents if possible for vddk input
by Martin Kletzander
It does not need configuring and speeds up the process of requesting extents if
the client asks for them one by one (like qemu-img).
Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com>
---
Essentially a v2 of the following patch since now there is a probing function:
https://www.redhat.com/archives/libguestfs/2019-September/msg00249.html
v2v/nbdkit.ml | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/v2v/nbdkit.ml b/v2v/nbdkit.ml
index 89fca337cbaa..77d2a506dc41 100644
--- a/v2v/nbdkit.ml
+++ b/v2v/nbdkit.ml
@@ -169,6 +169,13 @@ let common_create ?bandwidth plugin_name plugin_args plugin_env =
)
else [] in
+ (* Caching extents speeds up qemu-img, especially its consecutive
+ block_status requests with req_one=1.
+ *)
+ if probe_filter "cacheextents" then (
+ add_arg "--filter"; add_arg "cacheextents"
+ );
+
(* Retry filter (if it exists) can be used to get around brief
* interruptions in service. It must be closest to the plugin.
*)
--
2.24.0
4 years, 6 months