January 2018 - Libguestfs - libguestfs List Archives

Re: [Libguestfs] virt-resize changing cachedir
by Richard W.M. Jones 23 Jan '18

23 Jan '18

On Tue, Jan 23, 2018 at 06:01:08AM +0000, Ryan Lindsay wrote: > Hi Richard > > I have been playing around with your lovely libguestfs tools . I have however run into a bit of a problem > > Basically I have a 5.4T qcow2 virtual disk, which I made too small (bugger) > > So I had read that you can expand these with your virt-resize tools. > > So I tried this sort of thing > > virt-resize --expand /dev/sda1 virtual.qcow2 outdisk.qcow2 > > however it gives me the rather intersting Fatal error: exception Guestfs.Error("febootstrap-supermin-helper exited with error status 1, see debug messages above") > > Which I took to be non-optimal > > I believe however this could mean that my cachedir is to piddling in size > > > [root@space space]# virt-resize --expand /dev/sda1 virtual.qcow2 outdisk.qcow2 > Examining virtual.qcow2 ... > libguestfs: trace: set_verbose true > libguestfs: trace: set_verbose = 0 > libguestfs: create: flags = 0, handle = 0x1f827f0 > libguestfs: trace: add_drive "virtual.qcow2" "readonly:true" > libguestfs: trace: add_drive = 0 > libguestfs: trace: add_drive "outdisk.qcow2" "readonly:false" > libguestfs: trace: add_drive = 0 > libguestfs: trace: launch > libguestfs: trace: get_tmpdir > libguestfs: trace: get_tmpdir = "/tmp" > libguestfs: launch: attach-method=appliance > libguestfs: launch: tmpdir=/tmp/libguestfsteWT6W > libguestfs: launch: umask=0022 > libguestfs: launch: euid=0 > libguestfs: command: run: febootstrap-supermin-helper > libguestfs: command: run: \ --verbose > libguestfs: command: run: \ -f checksum > libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d > libguestfs: command: run: \ x86_64 > supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = (null), initrd = (null), appliance = (null) > supermin helper [00000ms] inputs[0] = /usr/lib64/guestfs/supermin.d > checking modpath /lib/modules/2.6.32-696.13.2.el6.x86_64 is a directory > picked vmlinuz-2.6.32-696.13.2.el6.x86_64 because modpath /lib/modules/2.6.32-696.13.2.el6.x86_64 exists > checking modpath /lib/modules/2.6.32-696.16.1.el6.x86_64 is a directory > picked vmlinuz-2.6.32-696.16.1.el6.x86_64 because modpath /lib/modules/2.6.32-696.16.1.el6.x86_64 exists > checking modpath /lib/modules/2.6.32-431.11.2.el6.x86_64 is a directory > picked vmlinuz-2.6.32-431.11.2.el6.x86_64 because modpath /lib/modules/2.6.32-431.11.2.el6.x86_64 exists > checking modpath /lib/modules/2.6.32-504.3.3.el6.x86_64 is a directory > picked vmlinuz-2.6.32-504.3.3.el6.x86_64 because modpath /lib/modules/2.6.32-504.3.3.el6.x86_64 exists > supermin helper [00000ms] finished creating kernel > supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d > supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/base.img > supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/daemon.img > supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/hostfiles > supermin helper [00022ms] visiting /usr/lib64/guestfs/supermin.d/init.img > supermin helper [00022ms] visiting /usr/lib64/guestfs/supermin.d/udev-rules.img > supermin helper [00022ms] adding kernel modules > supermin helper [00055ms] finished creating appliance > libguestfs: checksum of existing appliance: 9157ab378e1f61ccd23c368c72b4b1dd1131a5cbb309216a14de65c614223f62 > libguestfs: trace: get_cachedir > libguestfs: trace: get_cachedir = "/var/tmp" > libguestfs: trace: get_cachedir > libguestfs: trace: get_cachedir = "/var/tmp" > libguestfs: [00057ms] begin building supermin appliance > libguestfs: [00057ms] run supermin-helper > libguestfs: command: run: febootstrap-supermin-helper > libguestfs: command: run: \ --verbose > libguestfs: command: run: \ --copy-kernel > libguestfs: command: run: \ -f ext2 > libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d > libguestfs: command: run: \ x86_64 > libguestfs: command: run: \ /var/tmp/guestfs.S833dI/kernel > libguestfs: command: run: \ /var/tmp/guestfs.S833dI/initrd > libguestfs: command: run: \ /var/tmp/guestfs.S833dI/root > supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = /var/tmp/guestfs.S833dI/kernel, initrd = /var/tmp/guestfs.S833dI/initrd, appliance = /var/tmp/guestfs.S833dI/root > supermin helper [00000ms] inputs[0] = /usr/lib64/guestfs/supermin.d > checking modpath /lib/modules/2.6.32-696.13.2.el6.x86_64 is a directory > picked vmlinuz-2.6.32-696.13.2.el6.x86_64 because modpath /lib/modules/2.6.32-696.13.2.el6.x86_64 exists > checking modpath /lib/modules/2.6.32-696.16.1.el6.x86_64 is a directory > picked vmlinuz-2.6.32-696.16.1.el6.x86_64 because modpath /lib/modules/2.6.32-696.16.1.el6.x86_64 exists > checking modpath /lib/modules/2.6.32-431.11.2.el6.x86_64 is a directory > picked vmlinuz-2.6.32-431.11.2.el6.x86_64 because modpath /lib/modules/2.6.32-431.11.2.el6.x86_64 exists > checking modpath /lib/modules/2.6.32-504.3.3.el6.x86_64 is a directory > picked vmlinuz-2.6.32-504.3.3.el6.x86_64 because modpath /lib/modules/2.6.32-504.3.3.el6.x86_64 exists > supermin helper [00006ms] finished creating kernel > febootstrap-supermin-helper: open: /lib/modules/2.6.32-696.16.1.el6.x86_64/modules.dep: No such file or directory The error is this one. What happened to the modules.dep file? > libguestfs: command: run: rm > libguestfs: command: run: \ -rf /var/tmp/guestfs.S833dI > libguestfs: trace: launch = -1 (error) > Fatal error: exception Guestfs.Error("febootstrap-supermin-helper exited with error status 1, see debug messages above") > libguestfs: trace: close > libguestfs: closing guestfs handle 0x1f827f0 (state 0) > libguestfs: command: run: rm > libguestfs: command: run: \ -rf /tmp/libguestfsteWT6W > > So I guess my question is, can you tell virt-resize to use a different directory other than the paths you have set?? > > Is trying to extend a 5.4T disk a sensible thing to do? > > Are there other more interesting options? Yes, run ‘libguestfs-test-tool’ and attach the complete, unedited output. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

1 0

[PATCH] lua, perl: Use thread-safe strerror_r instead of strerror (RHBZ#1536763).
by Richard W.M. Jones 22 Jan '18

22 Jan '18

--- generator/lua.ml | 22 +++++++++++++++------- generator/perl.ml | 2 +- lua/Makefile.am | 1 + 3 files changed, 17 insertions(+), 8 deletions(-) diff --git a/generator/lua.ml b/generator/lua.ml index dd6aedbe9..7cfceb152 100644 --- a/generator/lua.ml +++ b/generator/lua.ml @@ -63,6 +63,8 @@ let generate_lua_c () = #endif #endif +#include \"ignore-value.h\" + #include <guestfs.h> #include \"guestfs-utils.h\" @@ -142,6 +144,7 @@ guestfs_int_lua_create (lua_State *L) guestfs_h *g; struct userdata *u; unsigned flags = 0; + char err[128]; if (lua_gettop (L) == 1) { OPTARG_IF_SET (1, \"environment\", @@ -157,9 +160,10 @@ guestfs_int_lua_create (lua_State *L) return luaL_error (L, \"Guestfs.create: too many arguments\"); g = guestfs_create_flags (flags); - if (!g) - return luaL_error (L, \"Guestfs.create: cannot create handle: %%s\", - strerror (errno)); + if (!g) { + ignore_value (strerror_r (errno, err, sizeof err)); + return luaL_error (L, \"Guestfs.create: cannot create handle: %%s\", err); + } guestfs_set_error_handler (g, NULL, NULL); @@ -226,6 +230,7 @@ error__tostring (lua_State *L) { int code; const char *msg; + char err[128]; lua_pushliteral (L, \"code\"); lua_gettable (L, 1); @@ -234,8 +239,10 @@ error__tostring (lua_State *L) lua_gettable (L, 1); msg = luaL_checkstring (L, -1); - if (code) - lua_pushfstring (L, \"%%s: %%s\", msg, strerror (code)); + if (code) { + ignore_value (strerror_r (code, err, sizeof err)); + lua_pushfstring (L, \"%%s: %%s\", msg, err); + } else lua_pushstring (L, msg); @@ -640,11 +647,12 @@ get_string_list (lua_State *L, int index) const size_t len = lua_objlen (L, index); size_t i; char **strs; + char err[128]; strs = malloc ((len+1) * sizeof (char *)); if (strs == NULL) { - luaL_error (L, \"get_string_list: malloc failed: %%s\", - strerror (errno)); + ignore_value (strerror_r (errno, err, sizeof err)); + luaL_error (L, \"get_string_list: malloc failed: %%s\", err); /*NOTREACHED*/ return NULL; } diff --git a/generator/perl.ml b/generator/perl.ml index bd7e328cd..240bf3b54 100644 --- a/generator/perl.ml +++ b/generator/perl.ml @@ -308,7 +308,7 @@ PREINIT: CODE: str = guestfs_event_to_string (event_bitmask); if (str == NULL) - croak (\"%%s\", strerror (errno)); + croak (\"%%m\"); RETVAL = newSVpv (str, 0); free (str); OUTPUT: diff --git a/lua/Makefile.am b/lua/Makefile.am index f90c1d7cb..a26c0baee 100644 --- a/lua/Makefile.am +++ b/lua/Makefile.am @@ -41,6 +41,7 @@ libluaguestfs_la_SOURCES = lua-guestfs.c libluaguestfs_la_CPPFLAGS = \ -DGUESTFS_PRIVATE=1 \ + -I$(srcdir)/../gnulib/lib -I../gnulib/lib \ -I$(top_srcdir)/common/utils -I$(top_builddir)/common/utils \ -I$(top_srcdir)/lib -I$(top_builddir)/lib -- 2.13.2

2 2

[PATCH] perl: Free array on error path to avoid leak (RHBZ#1536765).
by Richard W.M. Jones 22 Jan '18

22 Jan '18

--- generator/perl.ml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/generator/perl.ml b/generator/perl.ml index 240bf3b54..5fa00a92d 100644 --- a/generator/perl.ml +++ b/generator/perl.ml @@ -122,8 +122,10 @@ XS_unpack_charPtrPtr (SV *arg) { for (i = 0; i <= av_len (av); i++) { SV **elem = av_fetch (av, i, 0); - if (!elem || !*elem) + if (!elem || !*elem) { + free (ret); croak (\"missing element in list\"); + } ret[i] = SvPV_nolen (*elem); } -- 2.13.2

2 1

[PATCH nbdkit] filters: Add caching filter.
by Richard W.M. Jones 22 Jan '18

22 Jan '18

This adds a cache filter, which works like the COW filter in reverse. For realistic use it needs a bit more work, especially to add limits on the size of the cache, a more sensible cache replacement policy, and perhaps some kind of background worker to write dirty blocks out. Rich.

1 1

[PATCH nbdkit] filters: Add copy-on-write filter.
by Richard W.M. Jones 22 Jan '18

22 Jan '18

Eric, you'll probably find the design "interesting" ... It does work, for me at least. Rich.

2 6

[nbdkit PATCH v2 00/13] Add filters + FUA support to nbdkit
by Eric Blake 21 Jan '18

21 Jan '18

A combination of the work that both Rich and I have been doing lately, where filters use only the new API with flags on every command that the client can send over the wire (we can then add support for more flags in nbdkit without having to add new callbacks, as NBD adds more flags upstream). Eric Blake (4): protocol: Split flags from cmd field in requests backend: Pass flags argument through plugins: Move FUA fallback to plugins RFC: plugins: Add callbacks for FUA semantics Richard W.M. Jones (9): plugins: Move locking to a new file. Refactor plugin_* functions into a backend struct. Fix const-correctness of backend pwrite method. backend: Add a .plugin_name method. connections: Allow multiple handles to be stored in the connection object. Introduce filters. Implement filters. filters: Add nbdkit-offset-filter. filters: Move rdelay/wdelay from file plugin to new delay filter. Makefile.am | 2 +- TODO | 22 +- configure.ac | 5 +- docs/Makefile.am | 9 +- docs/nbdkit-filter.pod | 528 +++++++++++++++++++++++++ docs/nbdkit-plugin.pod | 92 ++++- docs/nbdkit.pod | 31 +- filters/Makefile.am | 35 ++ filters/delay/Makefile.am | 62 +++ filters/delay/delay.c | 162 ++++++++ filters/delay/nbdkit-delay-filter.pod | 88 +++++ filters/offset/Makefile.am | 62 +++ filters/offset/nbdkit-offset-filter.pod | 99 +++++ filters/offset/offset.c | 148 +++++++ include/Makefile.am | 4 +- include/nbdkit-filter.h | 147 +++++++ include/nbdkit-plugin.h | 33 +- nbdkit.in | 17 +- plugins/file/file.c | 76 +--- plugins/file/nbdkit-file-plugin.pod | 14 +- plugins/nbd/nbd.c | 39 +- src/Makefile.am | 7 +- src/connections.c | 124 +++--- src/filters.c | 613 ++++++++++++++++++++++++++++ src/internal.h | 92 +++-- src/locks.c | 115 ++++++ src/main.c | 134 ++++++- src/nbdkit.pc.in | 1 + src/plugins.c | 680 +++++++++++++++++--------------- src/protocol.h | 10 +- tests/test-parallel-file.sh | 4 +- tests/test-parallel-nbd.sh | 1 + 32 files changed, 2896 insertions(+), 560 deletions(-) create mode 100644 docs/nbdkit-filter.pod create mode 100644 filters/Makefile.am create mode 100644 filters/delay/Makefile.am create mode 100644 filters/delay/delay.c create mode 100644 filters/delay/nbdkit-delay-filter.pod create mode 100644 filters/offset/Makefile.am create mode 100644 filters/offset/nbdkit-offset-filter.pod create mode 100644 filters/offset/offset.c create mode 100644 include/nbdkit-filter.h create mode 100644 src/filters.c create mode 100644 src/locks.c -- 2.14.3

3 21

[nbdkit] A short note about filters in nbdkit
by Richard W.M. Jones 20 Jan '18

20 Jan '18

I've added support for filters to the nbdkit upstream repository (version >= 1.1.27). Filters allow you to modify plugins. A simple example would be selecting a partition from a plugin: nbdkit --filter=partition file file=disk.img partition=2 ^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^ which would serve the second partition only from disk.img. For more information see the nbdkit(1) and nbdkit-filter(3) man pages. For nbdkit plugins written in C we offer a long term stable ABI, where you can write a plugin (in C) now and have it work on all future versions of nbdkit. This means plugins can be distributed separately from nbdkit. However for nbdkit filters we haven't decided what, if any, ABI guarantees we will offer. The reason for this is simple: filters work by filtering plugin methods, but if we add new plugin methods then existing filters won't "see" the new methods. In some common cases this could even cause data corruption, for example if a filter is modifying file offsets like the partition filter above, then the new methods would write to the wrong part of the file. So the TL;DR is, for now, if you write a filter it is far better to submit it upstream so that we can keep it up to date. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

1 0

[PATCH nbdkit filters-v3 0/7] Introduce filters.
by Richard W.M. Jones 20 Jan '18

20 Jan '18

This is still tentative and needs a lot of work, but: - partition filter works, supporting MBR & GPT - prepare and finalize methods fixed - open method can now be changed (allowing readonly flag to be modified) - thread_model can be limited I believe I made most of the changes which were previously suggested in email. I think the only one I didn't was preventing inclusion of both <nbdkit-plugin.h> and <nbdkit-filter.h> in the same file. Rich.

2 10

[PATCH nbdkit filters-v2 0/5] Introduce filters.
by Richard W.M. Jones 19 Jan '18

19 Jan '18

Rebased filters patch. Requires current git master + the locks / thread model fix (https://www.redhat.com/archives/libguestfs/2018-January/msg00128.html) So a few changes here since last time: The "introduce filters" and "implement filters" patches are squashed together. I introduced a concept of .prepare and .finalize. These run before and after the data serving phase respectively. They're not a lot of use for plugins because you can just use .open and .close instead, and therefore I didn't expose them to plugins (only internally in the backend). However for filters they make more sense since they give you a place to prepare and finalize. They're necessary also for filters because filter.open could not call the next open method, save the (plugin) handle, then construct an nbdkit_next_ops object, and call pread. It _has_ to be split into two operations. This patch series is incomplete in at least three ways: - We still need a way for filter.open to call down to the next open function, so it can adjust the readonly flag (for copyonwrite filter, TBD). - Partition filter is obviously not complete. - Need to write a copyonwrite filter. Rich.

2 18

[PATCH] launch: direct: Use old-style file= and format= parameters when not disabling locking (RHBZ#1503497).
by Richard W.M. Jones 19 Jan '18

19 Jan '18

Ancient qemu 1.5 (in RHEL 7) does not understand the file.file.filename= and file.driver= parameters. Go back to using the old-style file= and format= parameters when we're not trying to set the file.backing.file.locking=off parameter. Fixes commit 9fe592808ccfd9ed184b88ca9c6cad2e1798dee3. Thanks: Yongkui Guo, Václav Kadlčík. --- lib/launch-direct.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/lib/launch-direct.c b/lib/launch-direct.c index aa4139359..47e8f37de 100644 --- a/lib/launch-direct.c +++ b/lib/launch-direct.c @@ -255,17 +255,28 @@ add_drive_standard_params (guestfs_h *g, struct backend_direct_data *data, } else { /* Writable qcow2 overlay on top of read-only drive. */ - append_list_format ("file.file.filename=%s", drv->overlay); - append_list ("file.driver=qcow2"); - append_list ("cache=unsafe"); - if (drv->disk_label) - append_list_format ("serial=%s", drv->disk_label); - - /* Add the file-specific locking option only for files, as qemu - * won't accept options unknown to the block driver in use. - */ - if (data->qemu_mandatory_locking && drv->src.protocol == drive_protocol_file) + if (data->qemu_mandatory_locking && + /* Add the file-specific locking option only for files, as + * qemu won't accept options unknown to the block driver in + * use. + */ + drv->src.protocol == drive_protocol_file) { + append_list_format ("file.file.filename=%s", drv->overlay); + append_list ("file.driver=qcow2"); append_list ("file.backing.file.locking=off"); + } + else { + /* Ancient qemu (esp. qemu 1.5 in RHEL 7) didn't understand the + * file.file.filename= parameter, so use the safer old-style + * form of parameters unless we actually want to specify the + * locking flag above. + */ + append_list_format ("file=%s", drv->overlay); + append_list ("format=qcow2"); + } + append_list ("cache=unsafe"); + if (drv->disk_label) + append_list_format ("serial=%s", drv->disk_label); } append_list_format ("id=hd%zu", i); -- 2.13.2

2 1