July 2019 - Libguestfs - libguestfs List Archives

[PATCH v2] launch: add support for autodetection of appliance image format
by Pavel Butsykin 06 Feb '20

06 Feb '20

This feature allows you to use different image formats for the fixed appliance. The raw format is used by default. Signed-off-by: Pavel Butsykin <pbutsykin(a)virtuozzo.com> --- lib/launch-direct.c | 2 ++ lib/launch-libvirt.c | 19 ++++++++++++------- m4/guestfs_appliance.m4 | 11 +++++++++++ 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index 0be662e25..b9b54857a 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -592,7 +592,9 @@ launch_direct (guestfs_h *g, void *datav, const char *arg) append_list ("id=appliance"); append_list ("cache=unsafe"); append_list ("if=none"); +#ifndef APPLIANCE_FMT_AUTO append_list ("format=raw"); +#endif } end_list (); start_list ("-device") { append_list ("scsi-hd"); diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c index 4adb2cfb3..030ea6911 100644 --- a/lib/launch-libvirt.c +++ b/lib/launch-libvirt.c @@ -212,9 +212,10 @@ get_source_format_or_autodetect (guestfs_h *g, struct drive *drv) /** * Create a qcow2 format overlay, with the given C<backing_drive> - * (file). The C<format> parameter, which must be non-NULL, is the - * backing file format. This is used to create the appliance overlay, - * and also for read-only drives. + * (file). The C<format> parameter is the backing file format. + * The C<format> parameter can be NULL, in this case the backing + * format will be determined automatically. This is used to create + * the appliance overlay, and also for read-only drives. */ static char * make_qcow2_overlay (guestfs_h *g, const char *backing_drive, @@ -223,8 +224,6 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, char *overlay; struct guestfs_disk_create_argv optargs; - assert (format != NULL); - if (guestfs_int_lazy_make_tmpdir (g) == -1) return NULL; @@ -232,8 +231,10 @@ make_qcow2_overlay (guestfs_h *g, const char *backing_drive, optargs.bitmask = GUESTFS_DISK_CREATE_BACKINGFILE_BITMASK; optargs.backingfile = backing_drive; - optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; - optargs.backingformat = format; + if (format) { + optargs.bitmask |= GUESTFS_DISK_CREATE_BACKINGFORMAT_BITMASK; + optargs.backingformat = format; + } if (guestfs_disk_create_argv (g, overlay, "qcow2", -1, &optargs) == -1) { free (overlay); @@ -461,7 +462,11 @@ launch_libvirt (guestfs_h *g, void *datav, const char *libvirt_uri) /* Note that appliance can be NULL if using the old-style appliance. */ if (appliance) { +#ifdef APPLIANCE_FMT_AUTO + params.appliance_overlay = make_qcow2_overlay (g, appliance, NULL); +#else params.appliance_overlay = make_qcow2_overlay (g, appliance, "raw"); +#endif if (!params.appliance_overlay) goto cleanup; } diff --git a/m4/guestfs_appliance.m4 b/m4/guestfs_appliance.m4 index 81c43879f..4e1ec8135 100644 --- a/m4/guestfs_appliance.m4 +++ b/m4/guestfs_appliance.m4 @@ -139,3 +139,14 @@ AC_SUBST([GUESTFS_DEFAULT_PATH]) AC_DEFINE_UNQUOTED([GUESTFS_DEFAULT_PATH], ["$GUESTFS_DEFAULT_PATH"], [Define guestfs default path.]) + +AC_ARG_ENABLE([appliance-fmt-auto], + [AS_HELP_STRING([--enable-appliance-fmt-auto], + [enable autodetection of appliance image format @<:@default=no@:>@])], + [ENABLE_APPLIANCE_FMT_AUTO="$enableval"], + [ENABLE_APPLIANCE_FMT_AUTO=no]) + +if test "x$ENABLE_APPLIANCE_FMT_AUTO" = "xyes"; then + AC_DEFINE([APPLIANCE_FMT_AUTO], [1], + [Define to 1 if enabled autodetection of appliance image format.]) +fi -- 2.13.0

2 4

1.39 proposal: Let's split up the libguestfs git repo and tarballs
by Richard W.M. Jones 29 Nov '19

29 Nov '19

My contention is that the libguestfs git repository is too large and unwieldy. There are too many separate, unrelated projects and as a result of that the source has too many dependencies and takes too long to build and test. The project divides (sort of) naturally into layers -- the library, the bindings, the various virt tools -- and could be split along those lines into separate projects which can then be released and evolve at their own pace. My suggested split would be something like this: * libguestfs: The library, daemon and appliance. That would include the following directories in a single project: appliance bash contrib daemon docs examples gnulib lib logo test-tool tmp utils website * 1 project for each language binding: csharp erlang gobject golang haskell java lua ocaml php perl python ruby * virt-customize and related tools, we'd probably call this subproject "virt-builder". It would include virt-builder, virt-customize and virt-sysprep, since they share a lot of common code. * 1 project for each of the following items: small tools written in C (virt-cat, virt-filesystems, virt-log, virt-ls, virt-tail, virt-diff, virt-edit, virt-format, guestmount, virt-inspector, virt-make-fs, virt-rescue) guestfish virt-alignment-scan and virt-df virt-dib virt-get-kernel virt-resize virt-sparsify virt-v2v and virt-p2v virt-win-reg * I'd be inclined to drop the legacy Perl tools virt-tar, virt-list-filesystems, virt-list-partitions unless someone especially wished to step forward to maintain them. * common code and generator: Off to the side we'd somehow need to package up the common code and the generator for use by all of the above projects. It wouldn't be a separate project for downstream packagers, but instead the code would be included (ie. duplicated) in tarballs and upstream available as a side git repo that you'd need to include when building (git submodule?). This is somewhat unspecified. M4, PO, and tests would be split between the projects as appropriate. My proposal would be to do this incrementally, rather than all at once, moving the easier things out first. Thoughts? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

4 18

[PATCH 0/6] p2v: start making it independent
by Pino Toscano 21 Aug '19

21 Aug '19

As preliminary steps in splitting virt-p2v to an own repository, start making p2v more independent within libguestfs. This is accomplished by the following changes: - generate the p2v kernel config sources & docs at build time using a Perl script, rather than the generator (so no need for OCaml when building from git, and no generated sources in dist tarballs) - create two local test images: instead of a phony Windows image there is a real Fedora guest, to avoid requiring hivex (for Windows) or more copied phony test data (for Fedora) - create an own .gitignore for p2v This is still not complete, although it does a number of important changes, and it still makes p2v usable within libguestfs. Pino Toscano (6): p2v: move kernel config to perl script p2v: move kernel config POD docs to perl script generator: remove p2v_config p2v: split gitignore p2v: tests: switch windows image with local fedora one p2v: tests: use a local blank-part disk image .gitignore | 37 -- configure.ac | 5 + docs/C_SOURCE_FILES | 3 - generator/Makefile.am | 3 - generator/main.ml | 9 - generator/p2v_config.ml | 835 -------------------------------- generator/p2v_config.mli | 22 - p2v/.gitignore | 48 ++ p2v/Makefile.am | 36 +- p2v/generate-p2v-config.pl | 915 ++++++++++++++++++++++++++++++++++++ p2v/test-virt-p2v-nbdkit.sh | 18 +- p2v/test-virt-p2v-pxe.sh | 16 +- p2v/test-virt-p2v.sh | 18 +- 13 files changed, 1014 insertions(+), 951 deletions(-) delete mode 100644 generator/p2v_config.ml delete mode 100644 generator/p2v_config.mli create mode 100644 p2v/.gitignore create mode 100755 p2v/generate-p2v-config.pl -- 2.21.0

2 9

[PATCH] v2v: rhv-upload-plugin - improve wait logic after finalize (RHBZ#1680361)
by Daniel Erez 09 Aug '19

09 Aug '19

After invoking transfer_service.finalize, check operation status by examining ImageTransferPhase and DiskStatus. This is done instead of failing after a predefined timeout regardless the status. * not verified * Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1680361 --- v2v/rhv-upload-plugin.py | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py index 2a950c5ed..873c11ce1 100644 --- a/v2v/rhv-upload-plugin.py +++ b/v2v/rhv-upload-plugin.py @@ -523,14 +523,30 @@ def close(h): # waiting for the transfer object to cease to exist, which # falls through to the exception case and then we can # continue. - endt = time.time() + timeout + start = time.time() try: while True: time.sleep(1) - tmp = transfer_service.get() - if time.time() > endt: - raise RuntimeError("timed out waiting for transfer " - "to finalize") + transfer = transfer_service.get() + + if transfer is None: + disk_service = h['disk_service'] + disk = disk_service.get() + if disk.status == types.DiskStatus.OK: + continue + + if transfer.phase == types.ImageTransferPhase.FINISHED_SUCCESS: + debug("finalized after %s seconds", time.time() - start) + break + + if transfer.phase == types.ImageTransferPhase.FINALIZING_SUCCESS: + if time.time() > start + timeout: + raise RuntimeError("timed out waiting for transfer " + "to finalize") + continue + + raise RuntimeError("Unexpected transfer phase while finalizing " + "upload %r" % transfer.phase) except sdk.NotFoundError: pass --

3 9

[PATCH] Rust bindings: Implement Event features
by Hiroyuki Katsura 06 Aug '19

06 Aug '19

This patch includes: - Event callback handlers - Tests related to events(410-430) --- generator/rust.ml | 38 ++++++- rust/src/base.rs | 24 +++-- rust/src/error.rs | 8 +- rust/src/event.rs | 158 ++++++++++++++++++++++++++++ rust/src/lib.rs | 2 + rust/tests/040_create_multiple.rs | 2 +- rust/tests/410_close_event.rs | 38 +++++++ rust/tests/420_log_messages.rs | 60 +++++++++++ rust/tests/430_progress_messages.rs | 61 +++++++++++ 9 files changed, 381 insertions(+), 10 deletions(-) create mode 100644 rust/src/event.rs create mode 100644 rust/tests/410_close_event.rs create mode 100644 rust/tests/420_log_messages.rs create mode 100644 rust/tests/430_progress_messages.rs diff --git a/generator/rust.ml b/generator/rust.ml index a1735602c..1f5cefa62 100644 --- a/generator/rust.ml +++ b/generator/rust.ml @@ -72,6 +72,42 @@ extern \"C\" { } "; + (* event enum *) + pr "\n"; + pr "pub enum Event {\n"; + List.iter ( + fun (name, _) -> + pr " %s,\n" (snake2caml name) + ) events; + pr "}\n\n"; + + pr "impl Event {\n"; + pr " pub fn to_u64(&self) -> u64 {\n"; + pr " match self {\n"; + List.iter ( + fun (name, i) -> + pr " Event::%s => %d,\n" (snake2caml name) i + ) events; + pr " }\n"; + pr " }\n"; + pr " pub fn from_bitmask(bitmask: u64) -> Option<Event> {\n"; + pr " match bitmask {\n"; + List.iter ( + fun (name, i) -> + pr " %d => Some(Event::%s),\n" i (snake2caml name) + ) events; + pr " _ => None,\n"; + pr " }\n"; + pr " }\n"; + pr "}\n\n"; + + pr "pub const EVENT_ALL: [Event; %d] = [\n" (List.length events); + List.iter ( + fun (name, _) -> + pr " Event::%s,\n" (snake2caml name) + ) events; + pr "];\n"; + List.iter ( fun { s_camel_name = name; s_name = c_name; s_cols = cols } -> pr "\n"; @@ -356,7 +392,7 @@ extern \"C\" { pr "}\n"; - pr "impl Handle {\n"; + pr "impl<'a> Handle<'a> {\n"; List.iter ( fun ({ name = name; shortdesc = shortdesc; longdesc = longdesc; style = (ret, args, optargs) } as f) -> diff --git a/rust/src/base.rs b/rust/src/base.rs index 02ad33535..0c6a3bdba 100644 --- a/rust/src/base.rs +++ b/rust/src/base.rs @@ -17,6 +17,10 @@ */ use crate::error; +use crate::event; +use crate::guestfs; +use std::collections; +use std::sync; #[allow(non_camel_case_types)] #[repr(C)] @@ -34,31 +38,37 @@ extern "C" { const GUESTFS_CREATE_NO_ENVIRONMENT: i64 = 1; const GUESTFS_CREATE_NO_CLOSE_ON_EXIT: i64 = 2; -pub struct Handle { +pub struct Handle<'a> { pub(crate) g: *mut guestfs_h, + pub(crate) callbacks: collections::HashMap< + event::EventHandle, + sync::Arc<Fn(guestfs::Event, event::EventHandle, &[u8], &[u64]) + 'a>, + >, } -impl Handle { - pub fn create() -> Result<Handle, error::Error> { +impl<'a> Handle<'a> { + pub fn create() -> Result<Handle<'a>, error::Error> { let g = unsafe { guestfs_create() }; if g.is_null() { Err(error::Error::Create) } else { - Ok(Handle { g }) + let callbacks = collections::HashMap::new(); + Ok(Handle { g, callbacks }) } } - pub fn create_flags(flags: CreateFlags) -> Result<Handle, error::Error> { + pub fn create_flags(flags: CreateFlags) -> Result<Handle<'a>, error::Error> { let g = unsafe { guestfs_create_flags(flags.to_libc_int()) }; if g.is_null() { Err(error::Error::Create) } else { - Ok(Handle { g }) + let callbacks = collections::HashMap::new(); + Ok(Handle { g, callbacks }) } } } -impl Drop for Handle { +impl<'a> Drop for Handle<'a> { fn drop(&mut self) { unsafe { guestfs_close(self.g) } } diff --git a/rust/src/error.rs b/rust/src/error.rs index 705ee1735..ce444e199 100644 --- a/rust/src/error.rs +++ b/rust/src/error.rs @@ -20,6 +20,7 @@ use crate::base; use crate::utils; use std::convert; use std::ffi; +use std::io; use std::os::raw::{c_char, c_int}; use std::str; @@ -41,6 +42,7 @@ pub enum Error { API(APIError), IllegalString(ffi::NulError), Utf8Error(str::Utf8Error), + UnixError(io::Error, &'static str), Create, } @@ -56,7 +58,11 @@ impl convert::From<str::Utf8Error> for Error { } } -impl base::Handle { +pub(crate) fn unix_error(operation: &'static str) -> Error { + Error::UnixError(io::Error::last_os_error(), operation) +} + +impl<'a> base::Handle<'a> { pub(crate) fn get_error_from_handle(&self, operation: &'static str) -> Error { let c_msg = unsafe { guestfs_last_error(self.g) }; let message = unsafe { utils::char_ptr_to_string(c_msg).unwrap() }; diff --git a/rust/src/event.rs b/rust/src/event.rs new file mode 100644 index 000000000..942feec69 --- /dev/null +++ b/rust/src/event.rs @@ -0,0 +1,158 @@ +/* libguestfs Rust bindings + * Copyright (C) 2019 Hiroyuki Katsura <hiroyuki.katsura.0513(a)gmail.com> + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +use crate::base; +use crate::error; +use crate::guestfs; +use std::ffi; +use std::os::raw::{c_char, c_void}; +use std::slice; +use std::sync; + +type GuestfsEventCallback = extern "C" fn( + *const base::guestfs_h, + *const c_void, + u64, + i32, + i32, + *const i8, + usize, + *const u64, + usize, +); + +#[link(name = "guestfs")] +extern "C" { + fn guestfs_set_event_callback( + g: *const base::guestfs_h, + cb: GuestfsEventCallback, + event_bitmask: u64, + flags: i32, + opaque: *const c_void, + ) -> i32; + fn guestfs_delete_event_callback(g: *const base::guestfs_h, eh: i32); + fn guestfs_event_to_string(bitmask: u64) -> *const c_char; + fn free(buf: *const c_void); +} + +#[derive(Hash, PartialEq, Eq)] +pub struct EventHandle { + eh: i32, +} + +pub type Callback = FnMut(guestfs::Event, EventHandle, &[i8], &[u64]); + +fn events_to_bitmask(v: &[guestfs::Event]) -> u64 { + let mut r = 0u64; + for x in v.iter() { + r |= x.to_u64(); + } + r +} + +pub fn event_to_string(events: &[guestfs::Event]) -> Result<String, error::Error> { + let bitmask = events_to_bitmask(events); + + let r = unsafe { guestfs_event_to_string(bitmask) }; + if r.is_null() { + Err(error::unix_error("event_to_string")) + } else { + let s = unsafe { ffi::CStr::from_ptr(r) }; + let s = s.to_str()?.to_string(); + unsafe { free(r as *const c_void) }; + Ok(s) + } +} + +/* -- Why Not Box<Callback> but Arc<Callback> (in struct base::Handle)? -- + * Assume that there are more than threads. While callback is running, + * if a thread frees the handle, automatically the buffer is freed if Box<Callback> + * is used. Therefore Arc<Callback> is used. + */ + +impl<'a> base::Handle<'a> { + pub fn set_event_callback<C: 'a>( + &mut self, + callback: C, + events: &[guestfs::Event], + ) -> Result<EventHandle, error::Error> + where + C: Fn(guestfs::Event, EventHandle, &[u8], &[u64]), + { + extern "C" fn trampoline<C>( + _g: *const base::guestfs_h, + opaque: *const c_void, + event: u64, + event_handle: i32, + _flags: i32, + buf: *const c_char, + buf_len: usize, + array: *const u64, + array_len: usize, + ) where + C: Fn(guestfs::Event, EventHandle, &[u8], &[u64]), + { + // trampoline function + // c.f. https://s3.amazonaws.com/temp.michaelfbryan.com/callbacks/index.html + + let event = match guestfs::Event::from_bitmask(event) { + Some(x) => x, + None => panic!("Failed to parse bitmask: {}", event), + }; + let eh = EventHandle { eh: event_handle }; + let buf = unsafe { slice::from_raw_parts(buf as *const u8, buf_len) }; + let array = unsafe { slice::from_raw_parts(array, array_len) }; + + let callback_ptr = unsafe { &*(opaque as *const sync::Arc<C>) }; + let callback = sync::Arc::clone(&callback_ptr); + callback(event, eh, buf, array) + } + let callback = sync::Arc::<C>::new(callback); + let event_bitmask = events_to_bitmask(events); + + let eh = { + // Weak::into_raw is nightly. + // In order to make sure that callback is freed when handle is freed, + // lifetime is explicitly declared. + let ptr: &'a sync::Arc<C> = Box::leak(Box::new(callback.clone())); + unsafe { + guestfs_set_event_callback( + self.g, + trampoline::<C>, + event_bitmask, + 0, + ptr as *const sync::Arc<C> as *const c_void, + ) + } + }; + if eh == -1 { + return Err(self.get_error_from_handle("set_event_callback")); + } + self.callbacks.insert(EventHandle { eh }, callback); + + Ok(EventHandle { eh }) + } + + pub fn delete_event_callback(&mut self, eh: EventHandle) -> Result<(), error::Error> { + unsafe { + guestfs_delete_event_callback(self.g, eh.eh); + } + self.callbacks.remove(&eh); + Ok(()) + } +} diff --git a/rust/src/lib.rs b/rust/src/lib.rs index cc41a99f8..81adef2a3 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -18,9 +18,11 @@ mod base; mod error; +mod event; mod guestfs; mod utils; pub use crate::base::*; pub use crate::error::*; +pub use crate::event::*; pub use crate::guestfs::*; diff --git a/rust/tests/040_create_multiple.rs b/rust/tests/040_create_multiple.rs index cc93554a3..970c988af 100644 --- a/rust/tests/040_create_multiple.rs +++ b/rust/tests/040_create_multiple.rs @@ -18,7 +18,7 @@ extern crate guestfs; -fn create() -> guestfs::Handle { +fn create<'a>() -> guestfs::Handle<'a> { match guestfs::Handle::create() { Ok(g) => g, Err(e) => panic!("fail: {:?}", e), diff --git a/rust/tests/410_close_event.rs b/rust/tests/410_close_event.rs new file mode 100644 index 000000000..308471098 --- /dev/null +++ b/rust/tests/410_close_event.rs @@ -0,0 +1,38 @@ +/* libguestfs Rust bindings + * Copyright (C) 2019 Hiroyuki Katsura <hiroyuki.katsura.0513(a)gmail.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +extern crate guestfs; + +use std::sync::{Arc, Mutex}; + +#[test] +fn close_event() { + let close_invoked = Arc::new(Mutex::new(0)); + { + let mut g = guestfs::Handle::create().expect("create"); + g.set_event_callback( + |_, _, _, _| { + let mut data = (&close_invoked).lock().unwrap(); + *data += 1; + }, + &[guestfs::Event::Close], + ) + .unwrap(); + } + assert_eq!(*((&close_invoked).lock().unwrap()), 1); +} diff --git a/rust/tests/420_log_messages.rs b/rust/tests/420_log_messages.rs new file mode 100644 index 000000000..1e9627ca7 --- /dev/null +++ b/rust/tests/420_log_messages.rs @@ -0,0 +1,60 @@ +/* libguestfs Rust bindings + * Copyright (C) 2019 Hiroyuki Katsura <hiroyuki.katsura.0513(a)gmail.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +extern crate guestfs; + +use std::str; +use std::sync::{Arc, Mutex}; + +#[test] +fn log_messages() { + let close_invoked = Arc::new(Mutex::new(0)); + { + let mut g = guestfs::Handle::create().expect("create"); + g.set_event_callback( + |ev, _, buf, array| { + let mut data = (&close_invoked).lock().unwrap(); + *data += 1; + + let event = guestfs::event_to_string(&[ev]).unwrap(); + + let buf = str::from_utf8(buf).unwrap(); + let array = array + .into_iter() + .map(|x| format!("{}", x)) + .collect::<Vec<String>>() + .join(","); + + eprintln!("event logged: event={} buf={} array={}", event, buf, array) + }, + &[ + guestfs::Event::Appliance, + guestfs::Event::Library, + guestfs::Event::Warning, + guestfs::Event::Trace, + ], + ) + .unwrap(); + + g.set_trace(true).unwrap(); + g.set_verbose(true).unwrap(); + g.add_drive_ro("/dev/null").unwrap(); + g.set_autosync(true).unwrap(); + } + assert!(*((&close_invoked).lock().unwrap()) > 0); +} diff --git a/rust/tests/430_progress_messages.rs b/rust/tests/430_progress_messages.rs new file mode 100644 index 000000000..a1d33aff7 --- /dev/null +++ b/rust/tests/430_progress_messages.rs @@ -0,0 +1,61 @@ +/* libguestfs Rust bindings + * Copyright (C) 2019 Hiroyuki Katsura <hiroyuki.katsura.0513(a)gmail.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +extern crate guestfs; + +use std::default::Default; +use std::sync::{Arc, Mutex}; + +#[test] +fn progress_messages() { + let callback_invoked = Arc::new(Mutex::new(0)); + { + let mut g = guestfs::Handle::create().expect("create"); + g.add_drive("/dev/null", Default::default()).unwrap(); + g.launch().unwrap(); + + let eh = g + .set_event_callback( + |_, _, _, _| { + let mut data = (&callback_invoked).lock().unwrap(); + *data += 1; + }, + &[guestfs::Event::Progress], + ) + .unwrap(); + assert_eq!("ok", g.debug("progress", &["5"]).unwrap()); + assert!(*(&callback_invoked).lock().unwrap() > 0); + + *(&callback_invoked).lock().unwrap() = 0; + g.delete_event_callback(eh).unwrap(); + assert_eq!("ok", g.debug("progress", &["5"]).unwrap()); + assert_eq!(*(&callback_invoked).lock().unwrap(), 0); + + g.set_event_callback( + |_, _, _, _| { + let mut data = (&callback_invoked).lock().unwrap(); + *data += 1; + }, + &[guestfs::Event::Progress], + ) + .unwrap(); + assert_eq!("ok", g.debug("progress", &["5"]).unwrap()); + assert!(*(&callback_invoked).lock().unwrap() > 0); + } + assert!(*(&callback_invoked).lock().unwrap() > 0); +} -- 2.20.1 (Apple Git-117)

5 13

[nbdkit PATCH 0/8] fd leak safety
by Eric Blake 01 Aug '19

01 Aug '19

There's enough here to need a review; some of it probably needs backporting to stable-1.12. This probably breaks tests on Haiku or other platforms that have not been as on-the-ball about atomic CLOEXEC; feel free to report issues that arise, and I'll help come up with workarounds (even if we end up leaving a rare fd leak on less-capable systems). Meanwhile, I'm still working on my promise to add an nbdkit_nanosleep for use in the delay and stat filters, and which makes nbdkit more responsive to ^C where it currently waits for a full delay to expire before exiting. (Nothing like finding several other pre-requisite bugs to fix first before getting to my real goal...) Eric Blake (8): rate: Pass through delay failures server: Don't leave uninit variable on failure server: Add test for nbdkit_read_password Revert "RHEL 5: Define O_CLOEXEC and SOCK_CLOEXEC." cow, cache: Better mkostemp fallback server: Atomically set CLOEXEC on all fds filters: Set CLOEXEC on files opened during .config rate: Atomically set CLOEXEC on fds server/internal.h | 8 ---- filters/cache/blk.c | 19 ++++++++- filters/cow/blk.c | 19 ++++++++- filters/log/log.c | 20 ++++++++- filters/rate/rate.c | 26 ++++++++---- filters/stats/stats.c | 18 +++++++- filters/xz/xzfile.c | 4 -- plugins/example2/example2.c | 4 -- plugins/file/file.c | 4 -- plugins/streaming/streaming.c | 4 -- server/connections.c | 1 + server/quit.c | 5 ++- server/sockets.c | 7 ++-- server/test-utils.c | 79 +++++++++++++++++++++++++++++++++-- server/utils.c | 10 ++++- 15 files changed, 182 insertions(+), 46 deletions(-) -- 2.20.1

3 20

[PATCH] v2v: -i vmx: Use scp -T option if available to unbreak scp (RHBZ#1733168).
by Richard W.M. Jones 01 Aug '19

01 Aug '19

Tested using: cd v2v LIBGUESTFS_BACKEND=direct ../run virt-v2v -i vmx -it ssh "ssh://localhost/$PWD/test-v2v-i-vmx-1.vmx" -o null -v -x and manually examining the debug output. Thanks: Ming Xie, Jakub Jelen. --- v2v/input_vmx.ml | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/v2v/input_vmx.ml b/v2v/input_vmx.ml index 5441bccb9..4a82a867f 100644 --- a/v2v/input_vmx.ml +++ b/v2v/input_vmx.ml @@ -61,6 +61,13 @@ let server_of_uri { Xml.uri_server } = let path_of_uri { Xml.uri_path } = match uri_path with None -> assert false | Some p -> p +let scp_supports_T_option = lazy ( + let cmd = "LANG=C scp -T |& grep \"unknown option\"" in + if verbose () then + eprintf "%s\n%!" cmd; + Sys.command cmd <> 0 +) + (* 'scp' a remote file into a temporary local file, returning the path * of the temporary local file. *) @@ -68,8 +75,9 @@ let scp_from_remote_to_temporary uri tmpdir filename = let localfile = tmpdir // filename in let cmd = - sprintf "scp%s%s %s%s:%s %s" + sprintf "scp%s%s%s %s%s:%s %s" (if verbose () then "" else " -q") + (if Lazy.force scp_supports_T_option then " -T" else "") (match port_of_uri uri with | None -> "" | Some port -> sprintf " -P %d" port) -- 2.22.0

2 1

[nbdkit PATCH v2] tests: Accommodate qemu-img 4.1 output change
by Eric Blake 30 Jul '19

30 Jul '19

Where qemu-img 4.0 used to say 'virtual size: 100M', the 4.1 release now says 'virtual size: 100 MiB'. Similarly, '5.0G' turned into '5 GiB'. But rather than worry about potential future changes to the human-readable output, we can just use --output=json (at which point we no longer even have to force qemu-img to the C locale). It might be slightly more robust to find our specific information using jq, but for now a grep of the output json is still reliable enough for our needs. If the '\b' is a problem on BSD, we could use '([, ]|$)' instead. Or that's where jq would make it easier to parse off a given number without worrying about what comes after the number. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- We could still use jq if desired, but it was easy enough to let these tests pass instead of skip when jq is not present. tests/test-ip.sh | 10 ++++++---- tests/test-nbd-tls-psk.sh | 6 +++--- tests/test-nbd-tls.sh | 6 +++--- tests/test-tls-psk.sh | 7 +++---- tests/test-tls.sh | 7 +++---- tests/test-truncate3.sh | 4 ++-- 6 files changed, 20 insertions(+), 20 deletions(-) diff --git a/tests/test-ip.sh b/tests/test-ip.sh index 636d3d3f..5a00a2ec 100755 --- a/tests/test-ip.sh +++ b/tests/test-ip.sh @@ -57,15 +57,17 @@ kill -s 0 $pid # Check we can connect over the IPv4 loopback interface. ipv4_lo="$(ip -o -4 addr show scope host)" if test -n "$ipv4_lo"; then - qemu-img info --image-opts "file.driver=nbd,file.host=127.0.0.1,file.port=$port" > ipv4.out + qemu-img info --output=json \ + --image-opts "file.driver=nbd,file.host=127.0.0.1,file.port=$port" > ipv4.out cat ipv4.out - grep -sq "^virtual size: 100M" ipv4.out + grep -sq '"virtual-size": *104857600\b' ipv4.out fi # Check we can connect over the IPv6 loopback interface. ipv6_lo="$(ip -o -6 addr show scope host)" if test -n "$ipv6_lo"; then - qemu-img info --image-opts "file.driver=nbd,file.host=::1,file.port=$port" > ipv6.out + qemu-img info --output=json \ + --image-opts "file.driver=nbd,file.host=::1,file.port=$port" > ipv6.out cat ipv6.out - grep -sq "^virtual size: 100M" ipv6.out + grep -sq '"virtual-size": *104857600\b' ipv6.out fi diff --git a/tests/test-nbd-tls-psk.sh b/tests/test-nbd-tls-psk.sh index d0bbc468..82822d11 100755 --- a/tests/test-nbd-tls-psk.sh +++ b/tests/test-nbd-tls-psk.sh @@ -73,9 +73,9 @@ LIBNBD_DEBUG=1 start_nbdkit -P "$pid2" -U "$sock2" --tls=off \ nbd tls=require tls-psk=keys.psk tls-username=qemu socket="$sock1" # Run unencrypted client -LANG=C qemu-img info -f raw "nbd+unix:///?socket=$sock2" > nbd-tls-psk.out +qemu-img info --output=json -f raw "nbd+unix:///?socket=$sock2" > nbd-tls-psk.out cat nbd-tls-psk.out -grep -sq "^file format: raw" nbd-tls-psk.out -grep -sq "^virtual size: 100M" nbd-tls-psk.out +grep -sq '"format": *"raw"' nbd-tls-psk.out +grep -sq '"virtual-size": *104857600\b' nbd-tls-psk.out diff --git a/tests/test-nbd-tls.sh b/tests/test-nbd-tls.sh index af824d23..c4f4faca 100755 --- a/tests/test-nbd-tls.sh +++ b/tests/test-nbd-tls.sh @@ -74,9 +74,9 @@ LIBNBD_DEBUG=1 start_nbdkit -P "$pid2" -U "$sock2" --tls=off \ nbd tls=require tls-certificates="$pkidir" socket="$sock1" # Run unencrypted client -LANG=C qemu-img info -f raw "nbd+unix:///?socket=$sock2" > nbd-tls.out +qemu-img info --output=json -f raw "nbd+unix:///?socket=$sock2" > nbd-tls.out cat nbd-tls.out -grep -sq "^file format: raw" nbd-tls.out -grep -sq "^virtual size: 100M" nbd-tls.out +grep -sq '"format": *"raw"' nbd-tls.out +grep -sq '"virtual-size": *104857600\b' nbd-tls.out diff --git a/tests/test-tls-psk.sh b/tests/test-tls-psk.sh index 393f5893..a4ecb604 100755 --- a/tests/test-tls-psk.sh +++ b/tests/test-tls-psk.sh @@ -72,12 +72,11 @@ start_nbdkit -P tls-psk.pid -p $port -n \ --tls=require --tls-psk=keys.psk example1 # Run qemu-img against the server. -LANG=C \ -qemu-img info \ +qemu-img info --output=json \ --object "tls-creds-psk,id=tls0,endpoint=client,dir=$PWD" \ --image-opts "file.driver=nbd,file.host=localhost,file.port=$port,file.tls-creds=tls0" > tls-psk.out cat tls-psk.out -grep -sq "^file format: raw" tls-psk.out -grep -sq "^virtual size: 100M" tls-psk.out +grep -sq '"format": *"raw"' tls-psk.out +grep -sq '"virtual-size": *104857600\b' tls-psk.out diff --git a/tests/test-tls.sh b/tests/test-tls.sh index 70d40aea..2718e552 100755 --- a/tests/test-tls.sh +++ b/tests/test-tls.sh @@ -65,12 +65,11 @@ start_nbdkit -P tls.pid -p $port -n --tls=require \ --tls-certificates="$pkidir" example1 # Run qemu-img against the server. -LANG=C \ -qemu-img info \ +qemu-img info --output=json \ --object "tls-creds-x509,id=tls0,endpoint=client,dir=$pkidir" \ --image-opts "file.driver=nbd,file.host=localhost,file.port=$port,file.tls-creds=tls0" > tls.out cat tls.out -grep -sq "^file format: raw" tls.out -grep -sq "^virtual size: 100M" tls.out +grep -sq '"format": *"raw"' tls.out +grep -sq '"virtual-size": *104857600\b' tls.out diff --git a/tests/test-truncate3.sh b/tests/test-truncate3.sh index 0a7fba8b..885d58f0 100755 --- a/tests/test-truncate3.sh +++ b/tests/test-truncate3.sh @@ -50,8 +50,8 @@ start_nbdkit -P truncate3.pid -U $sock \ pattern 5G \ round-up=512 -LANG=C qemu-img info nbd:unix:$sock > truncate3.out -if ! grep "virtual size: 5.0G" truncate3.out; then +qemu-img info --output=json nbd:unix:$sock > truncate3.out +if ! grep '"virtual-size": *5368709120' truncate3.out; then echo "$0: unexpected output from truncate3 regression test:" cat truncate3.out exit 1 -- 2.20.1

2 1

[nbdkit PATCH] tests: Accommodate qemu-img 4.1 output change
by Eric Blake 30 Jul '19

30 Jul '19

Where qemu-img 4.0 used to say 'virtual size: 100M', the 4.1 release now says 'virtual size: 100 MiB'. Similarly, '5.0G' turned into '5 GiB'. Update expected test output to tolerate either version of qemu. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- I already know I want to send a v2; on IRC, Rich pointed out that 'qemu-img info --output=json' is less prone to fickle changes, and we already depend on jq elsewhere in the testsuite. But since I'd already got this written up, I'm at least posting it (if nothing else, to have a list archive to point to when someone else complains about qemu-img changing output). tests/test-ip.sh | 4 ++-- tests/test-nbd-tls-psk.sh | 2 +- tests/test-nbd-tls.sh | 2 +- tests/test-tls-psk.sh | 2 +- tests/test-tls.sh | 2 +- tests/test-truncate3.sh | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/test-ip.sh b/tests/test-ip.sh index 636d3d3f..60f2e066 100755 --- a/tests/test-ip.sh +++ b/tests/test-ip.sh @@ -59,7 +59,7 @@ ipv4_lo="$(ip -o -4 addr show scope host)" if test -n "$ipv4_lo"; then qemu-img info --image-opts "file.driver=nbd,file.host=127.0.0.1,file.port=$port" > ipv4.out cat ipv4.out - grep -sq "^virtual size: 100M" ipv4.out + grep -sq "^virtual size: 100 *M" ipv4.out fi # Check we can connect over the IPv6 loopback interface. @@ -67,5 +67,5 @@ ipv6_lo="$(ip -o -6 addr show scope host)" if test -n "$ipv6_lo"; then qemu-img info --image-opts "file.driver=nbd,file.host=::1,file.port=$port" > ipv6.out cat ipv6.out - grep -sq "^virtual size: 100M" ipv6.out + grep -sq "^virtual size: 100 *M" ipv6.out fi diff --git a/tests/test-nbd-tls-psk.sh b/tests/test-nbd-tls-psk.sh index d0bbc468..e07d553b 100755 --- a/tests/test-nbd-tls-psk.sh +++ b/tests/test-nbd-tls-psk.sh @@ -78,4 +78,4 @@ LANG=C qemu-img info -f raw "nbd+unix:///?socket=$sock2" > nbd-tls-psk.out cat nbd-tls-psk.out grep -sq "^file format: raw" nbd-tls-psk.out -grep -sq "^virtual size: 100M" nbd-tls-psk.out +grep -sq "^virtual size: 100 *M" nbd-tls-psk.out diff --git a/tests/test-nbd-tls.sh b/tests/test-nbd-tls.sh index af824d23..11fdea22 100755 --- a/tests/test-nbd-tls.sh +++ b/tests/test-nbd-tls.sh @@ -79,4 +79,4 @@ LANG=C qemu-img info -f raw "nbd+unix:///?socket=$sock2" > nbd-tls.out cat nbd-tls.out grep -sq "^file format: raw" nbd-tls.out -grep -sq "^virtual size: 100M" nbd-tls.out +grep -sq "^virtual size: 100 *M" nbd-tls.out diff --git a/tests/test-tls-psk.sh b/tests/test-tls-psk.sh index 393f5893..c0f03487 100755 --- a/tests/test-tls-psk.sh +++ b/tests/test-tls-psk.sh @@ -80,4 +80,4 @@ qemu-img info \ cat tls-psk.out grep -sq "^file format: raw" tls-psk.out -grep -sq "^virtual size: 100M" tls-psk.out +grep -sq "^virtual size: 100 *M" tls-psk.out diff --git a/tests/test-tls.sh b/tests/test-tls.sh index 70d40aea..5936f9fa 100755 --- a/tests/test-tls.sh +++ b/tests/test-tls.sh @@ -73,4 +73,4 @@ qemu-img info \ cat tls.out grep -sq "^file format: raw" tls.out -grep -sq "^virtual size: 100M" tls.out +grep -sq "^virtual size: 100 *M" tls.out diff --git a/tests/test-truncate3.sh b/tests/test-truncate3.sh index 0a7fba8b..396fd253 100755 --- a/tests/test-truncate3.sh +++ b/tests/test-truncate3.sh @@ -51,7 +51,7 @@ start_nbdkit -P truncate3.pid -U $sock \ round-up=512 LANG=C qemu-img info nbd:unix:$sock > truncate3.out -if ! grep "virtual size: 5.0G" truncate3.out; then +if ! grep "virtual size:.*(5368709120 " truncate3.out; then echo "$0: unexpected output from truncate3 regression test:" cat truncate3.out exit 1 -- 2.20.1

2 1

[PATCH nbdkit] nbd: Update for libnbd 0.9.6.
by Richard W.M. Jones 30 Jul '19

30 Jul '19

--- configure.ac | 4 ++-- plugins/nbd/nbd.c | 12 +++++++++--- README | 2 +- 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/configure.ac b/configure.ac index 8c7ee8b..0ce78c7 100644 --- a/configure.ac +++ b/configure.ac @@ -718,12 +718,12 @@ AC_ARG_WITH([libnbd], [], [with_libnbd=check]) AS_IF([test "$with_libnbd" != "no"],[ - PKG_CHECK_MODULES([LIBNBD], [libnbd >= 0.1.9],[ + PKG_CHECK_MODULES([LIBNBD], [libnbd >= 0.9.6],[ AC_SUBST([LIBNBD_CFLAGS]) AC_SUBST([LIBNBD_LIBS]) AC_DEFINE([HAVE_LIBNBD],[1],[libnbd found at compile time.]) ], - [AC_MSG_WARN([libnbd >= 0.1.9 not found, nbd plugin will be crippled])]) + [AC_MSG_WARN([libnbd >= 0.9.6 not found, nbd plugin will be crippled])]) ]) AM_CONDITIONAL([HAVE_LIBNBD], [test "x$LIBNBD_LIBS" != "x"]) diff --git a/plugins/nbd/nbd.c b/plugins/nbd/nbd.c index 2ddb2c0..e8bc779 100644 --- a/plugins/nbd/nbd.c +++ b/plugins/nbd/nbd.c @@ -57,6 +57,7 @@ /* The per-transaction details */ struct transaction { + int64_t cookie; sem_t sem; uint32_t early_err; uint32_t err; @@ -353,15 +354,19 @@ nbdplug_prepare (struct transaction *trans) } static int -nbdplug_notify (unsigned valid_flag, void *opaque, int64_t cookie, int *error) +nbdplug_notify (unsigned valid_flag, void *opaque, int *error) { struct transaction *trans = opaque; if (!(valid_flag & LIBNBD_CALLBACK_VALID)) return 0; + /* There's a possible race here where trans->cookie has not yet been + * updated by nbdplug_register, but it's only an informational + * message. + */ nbdkit_debug ("cookie %" PRId64 " completed state machine, status %d", - cookie, *error); + trans->cookie, *error); trans->err = *error; if (sem_post (&trans->sem)) { nbdkit_error ("failed to post semaphore: %m"); @@ -383,6 +388,7 @@ nbdplug_register (struct handle *h, struct transaction *trans, int64_t cookie) } nbdkit_debug ("cookie %" PRId64 " started by state machine", cookie); + trans->cookie = cookie; if (write (h->fds[1], &c, 1) != 1 && errno != EAGAIN) nbdkit_debug ("failed to kick reader thread: %m"); @@ -504,7 +510,7 @@ nbdplug_open (int readonly) static void nbdplug_close_handle (struct handle *h) { - if (nbd_shutdown (h->nbd) == -1) + if (nbd_shutdown (h->nbd, 0) == -1) nbdkit_debug ("failed to clean up handle: %s", nbd_get_error ()); if ((errno = pthread_join (h->reader, NULL))) nbdkit_debug ("failed to join reader thread: %m"); diff --git a/README b/README index 26614d5..06c16dd 100644 --- a/README +++ b/README @@ -113,7 +113,7 @@ For the linuxdisk plugin: For the nbd plugin, to get URI and TLS support: - - libnbd >= 0.1.9 + - libnbd >= 0.9.6 For the Perl, example4 and tar plugins: -- 2.22.0

2 1