Re: [Libguestfs] virt-resize changing cachedir
by Richard W.M. Jones
On Tue, Jan 23, 2018 at 06:01:08AM +0000, Ryan Lindsay wrote:
> Hi Richard
>
> I have been playing around with your lovely libguestfs tools . I have however run into a bit of a problem
>
> Basically I have a 5.4T qcow2 virtual disk, which I made too small (bugger)
>
> So I had read that you can expand these with your virt-resize tools.
>
> So I tried this sort of thing
>
> virt-resize --expand /dev/sda1 virtual.qcow2 outdisk.qcow2
>
> however it gives me the rather intersting Fatal error: exception Guestfs.Error("febootstrap-supermin-helper exited with error status 1, see debug messages above")
>
> Which I took to be non-optimal
>
> I believe however this could mean that my cachedir is to piddling in size
>
>
> [root@space space]# virt-resize --expand /dev/sda1 virtual.qcow2 outdisk.qcow2
> Examining virtual.qcow2 ...
> libguestfs: trace: set_verbose true
> libguestfs: trace: set_verbose = 0
> libguestfs: create: flags = 0, handle = 0x1f827f0
> libguestfs: trace: add_drive "virtual.qcow2" "readonly:true"
> libguestfs: trace: add_drive = 0
> libguestfs: trace: add_drive "outdisk.qcow2" "readonly:false"
> libguestfs: trace: add_drive = 0
> libguestfs: trace: launch
> libguestfs: trace: get_tmpdir
> libguestfs: trace: get_tmpdir = "/tmp"
> libguestfs: launch: attach-method=appliance
> libguestfs: launch: tmpdir=/tmp/libguestfsteWT6W
> libguestfs: launch: umask=0022
> libguestfs: launch: euid=0
> libguestfs: command: run: febootstrap-supermin-helper
> libguestfs: command: run: \ --verbose
> libguestfs: command: run: \ -f checksum
> libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
> libguestfs: command: run: \ x86_64
> supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = (null), initrd = (null), appliance = (null)
> supermin helper [00000ms] inputs[0] = /usr/lib64/guestfs/supermin.d
> checking modpath /lib/modules/2.6.32-696.13.2.el6.x86_64 is a directory
> picked vmlinuz-2.6.32-696.13.2.el6.x86_64 because modpath /lib/modules/2.6.32-696.13.2.el6.x86_64 exists
> checking modpath /lib/modules/2.6.32-696.16.1.el6.x86_64 is a directory
> picked vmlinuz-2.6.32-696.16.1.el6.x86_64 because modpath /lib/modules/2.6.32-696.16.1.el6.x86_64 exists
> checking modpath /lib/modules/2.6.32-431.11.2.el6.x86_64 is a directory
> picked vmlinuz-2.6.32-431.11.2.el6.x86_64 because modpath /lib/modules/2.6.32-431.11.2.el6.x86_64 exists
> checking modpath /lib/modules/2.6.32-504.3.3.el6.x86_64 is a directory
> picked vmlinuz-2.6.32-504.3.3.el6.x86_64 because modpath /lib/modules/2.6.32-504.3.3.el6.x86_64 exists
> supermin helper [00000ms] finished creating kernel
> supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d
> supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/base.img
> supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/daemon.img
> supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/hostfiles
> supermin helper [00022ms] visiting /usr/lib64/guestfs/supermin.d/init.img
> supermin helper [00022ms] visiting /usr/lib64/guestfs/supermin.d/udev-rules.img
> supermin helper [00022ms] adding kernel modules
> supermin helper [00055ms] finished creating appliance
> libguestfs: checksum of existing appliance: 9157ab378e1f61ccd23c368c72b4b1dd1131a5cbb309216a14de65c614223f62
> libguestfs: trace: get_cachedir
> libguestfs: trace: get_cachedir = "/var/tmp"
> libguestfs: trace: get_cachedir
> libguestfs: trace: get_cachedir = "/var/tmp"
> libguestfs: [00057ms] begin building supermin appliance
> libguestfs: [00057ms] run supermin-helper
> libguestfs: command: run: febootstrap-supermin-helper
> libguestfs: command: run: \ --verbose
> libguestfs: command: run: \ --copy-kernel
> libguestfs: command: run: \ -f ext2
> libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
> libguestfs: command: run: \ x86_64
> libguestfs: command: run: \ /var/tmp/guestfs.S833dI/kernel
> libguestfs: command: run: \ /var/tmp/guestfs.S833dI/initrd
> libguestfs: command: run: \ /var/tmp/guestfs.S833dI/root
> supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = /var/tmp/guestfs.S833dI/kernel, initrd = /var/tmp/guestfs.S833dI/initrd, appliance = /var/tmp/guestfs.S833dI/root
> supermin helper [00000ms] inputs[0] = /usr/lib64/guestfs/supermin.d
> checking modpath /lib/modules/2.6.32-696.13.2.el6.x86_64 is a directory
> picked vmlinuz-2.6.32-696.13.2.el6.x86_64 because modpath /lib/modules/2.6.32-696.13.2.el6.x86_64 exists
> checking modpath /lib/modules/2.6.32-696.16.1.el6.x86_64 is a directory
> picked vmlinuz-2.6.32-696.16.1.el6.x86_64 because modpath /lib/modules/2.6.32-696.16.1.el6.x86_64 exists
> checking modpath /lib/modules/2.6.32-431.11.2.el6.x86_64 is a directory
> picked vmlinuz-2.6.32-431.11.2.el6.x86_64 because modpath /lib/modules/2.6.32-431.11.2.el6.x86_64 exists
> checking modpath /lib/modules/2.6.32-504.3.3.el6.x86_64 is a directory
> picked vmlinuz-2.6.32-504.3.3.el6.x86_64 because modpath /lib/modules/2.6.32-504.3.3.el6.x86_64 exists
> supermin helper [00006ms] finished creating kernel
> febootstrap-supermin-helper: open: /lib/modules/2.6.32-696.16.1.el6.x86_64/modules.dep: No such file or directory
The error is this one. What happened to the modules.dep file?
> libguestfs: command: run: rm
> libguestfs: command: run: \ -rf /var/tmp/guestfs.S833dI
> libguestfs: trace: launch = -1 (error)
> Fatal error: exception Guestfs.Error("febootstrap-supermin-helper exited with error status 1, see debug messages above")
> libguestfs: trace: close
> libguestfs: closing guestfs handle 0x1f827f0 (state 0)
> libguestfs: command: run: rm
> libguestfs: command: run: \ -rf /tmp/libguestfsteWT6W
>
> So I guess my question is, can you tell virt-resize to use a different directory other than the paths you have set??
>
> Is trying to extend a 5.4T disk a sensible thing to do?
>
> Are there other more interesting options?
Yes, run ‘libguestfs-test-tool’ and attach the complete, unedited
output.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
6 years, 9 months
[PATCH] lua, perl: Use thread-safe strerror_r instead of strerror (RHBZ#1536763).
by Richard W.M. Jones
---
generator/lua.ml | 22 +++++++++++++++-------
generator/perl.ml | 2 +-
lua/Makefile.am | 1 +
3 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/generator/lua.ml b/generator/lua.ml
index dd6aedbe9..7cfceb152 100644
--- a/generator/lua.ml
+++ b/generator/lua.ml
@@ -63,6 +63,8 @@ let generate_lua_c () =
#endif
#endif
+#include \"ignore-value.h\"
+
#include <guestfs.h>
#include \"guestfs-utils.h\"
@@ -142,6 +144,7 @@ guestfs_int_lua_create (lua_State *L)
guestfs_h *g;
struct userdata *u;
unsigned flags = 0;
+ char err[128];
if (lua_gettop (L) == 1) {
OPTARG_IF_SET (1, \"environment\",
@@ -157,9 +160,10 @@ guestfs_int_lua_create (lua_State *L)
return luaL_error (L, \"Guestfs.create: too many arguments\");
g = guestfs_create_flags (flags);
- if (!g)
- return luaL_error (L, \"Guestfs.create: cannot create handle: %%s\",
- strerror (errno));
+ if (!g) {
+ ignore_value (strerror_r (errno, err, sizeof err));
+ return luaL_error (L, \"Guestfs.create: cannot create handle: %%s\", err);
+ }
guestfs_set_error_handler (g, NULL, NULL);
@@ -226,6 +230,7 @@ error__tostring (lua_State *L)
{
int code;
const char *msg;
+ char err[128];
lua_pushliteral (L, \"code\");
lua_gettable (L, 1);
@@ -234,8 +239,10 @@ error__tostring (lua_State *L)
lua_gettable (L, 1);
msg = luaL_checkstring (L, -1);
- if (code)
- lua_pushfstring (L, \"%%s: %%s\", msg, strerror (code));
+ if (code) {
+ ignore_value (strerror_r (code, err, sizeof err));
+ lua_pushfstring (L, \"%%s: %%s\", msg, err);
+ }
else
lua_pushstring (L, msg);
@@ -640,11 +647,12 @@ get_string_list (lua_State *L, int index)
const size_t len = lua_objlen (L, index);
size_t i;
char **strs;
+ char err[128];
strs = malloc ((len+1) * sizeof (char *));
if (strs == NULL) {
- luaL_error (L, \"get_string_list: malloc failed: %%s\",
- strerror (errno));
+ ignore_value (strerror_r (errno, err, sizeof err));
+ luaL_error (L, \"get_string_list: malloc failed: %%s\", err);
/*NOTREACHED*/
return NULL;
}
diff --git a/generator/perl.ml b/generator/perl.ml
index bd7e328cd..240bf3b54 100644
--- a/generator/perl.ml
+++ b/generator/perl.ml
@@ -308,7 +308,7 @@ PREINIT:
CODE:
str = guestfs_event_to_string (event_bitmask);
if (str == NULL)
- croak (\"%%s\", strerror (errno));
+ croak (\"%%m\");
RETVAL = newSVpv (str, 0);
free (str);
OUTPUT:
diff --git a/lua/Makefile.am b/lua/Makefile.am
index f90c1d7cb..a26c0baee 100644
--- a/lua/Makefile.am
+++ b/lua/Makefile.am
@@ -41,6 +41,7 @@ libluaguestfs_la_SOURCES = lua-guestfs.c
libluaguestfs_la_CPPFLAGS = \
-DGUESTFS_PRIVATE=1 \
+ -I$(srcdir)/../gnulib/lib -I../gnulib/lib \
-I$(top_srcdir)/common/utils -I$(top_builddir)/common/utils \
-I$(top_srcdir)/lib -I$(top_builddir)/lib
--
2.13.2
6 years, 9 months
[PATCH] perl: Free array on error path to avoid leak (RHBZ#1536765).
by Richard W.M. Jones
---
generator/perl.ml | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/generator/perl.ml b/generator/perl.ml
index 240bf3b54..5fa00a92d 100644
--- a/generator/perl.ml
+++ b/generator/perl.ml
@@ -122,8 +122,10 @@ XS_unpack_charPtrPtr (SV *arg) {
for (i = 0; i <= av_len (av); i++) {
SV **elem = av_fetch (av, i, 0);
- if (!elem || !*elem)
+ if (!elem || !*elem) {
+ free (ret);
croak (\"missing element in list\");
+ }
ret[i] = SvPV_nolen (*elem);
}
--
2.13.2
6 years, 9 months
[PATCH nbdkit] filters: Add caching filter.
by Richard W.M. Jones
This adds a cache filter, which works like the COW filter in reverse.
For realistic use it needs a bit more work, especially to add limits
on the size of the cache, a more sensible cache replacement policy,
and perhaps some kind of background worker to write dirty blocks out.
Rich.
6 years, 9 months
[nbdkit PATCH v2 00/13] Add filters + FUA support to nbdkit
by Eric Blake
A combination of the work that both Rich and I have been doing
lately, where filters use only the new API with flags on every
command that the client can send over the wire (we can then
add support for more flags in nbdkit without having to add new
callbacks, as NBD adds more flags upstream).
Eric Blake (4):
protocol: Split flags from cmd field in requests
backend: Pass flags argument through
plugins: Move FUA fallback to plugins
RFC: plugins: Add callbacks for FUA semantics
Richard W.M. Jones (9):
plugins: Move locking to a new file.
Refactor plugin_* functions into a backend struct.
Fix const-correctness of backend pwrite method.
backend: Add a .plugin_name method.
connections: Allow multiple handles to be stored in the connection
object.
Introduce filters.
Implement filters.
filters: Add nbdkit-offset-filter.
filters: Move rdelay/wdelay from file plugin to new delay filter.
Makefile.am | 2 +-
TODO | 22 +-
configure.ac | 5 +-
docs/Makefile.am | 9 +-
docs/nbdkit-filter.pod | 528 +++++++++++++++++++++++++
docs/nbdkit-plugin.pod | 92 ++++-
docs/nbdkit.pod | 31 +-
filters/Makefile.am | 35 ++
filters/delay/Makefile.am | 62 +++
filters/delay/delay.c | 162 ++++++++
filters/delay/nbdkit-delay-filter.pod | 88 +++++
filters/offset/Makefile.am | 62 +++
filters/offset/nbdkit-offset-filter.pod | 99 +++++
filters/offset/offset.c | 148 +++++++
include/Makefile.am | 4 +-
include/nbdkit-filter.h | 147 +++++++
include/nbdkit-plugin.h | 33 +-
nbdkit.in | 17 +-
plugins/file/file.c | 76 +---
plugins/file/nbdkit-file-plugin.pod | 14 +-
plugins/nbd/nbd.c | 39 +-
src/Makefile.am | 7 +-
src/connections.c | 124 +++---
src/filters.c | 613 ++++++++++++++++++++++++++++
src/internal.h | 92 +++--
src/locks.c | 115 ++++++
src/main.c | 134 ++++++-
src/nbdkit.pc.in | 1 +
src/plugins.c | 680 +++++++++++++++++---------------
src/protocol.h | 10 +-
tests/test-parallel-file.sh | 4 +-
tests/test-parallel-nbd.sh | 1 +
32 files changed, 2896 insertions(+), 560 deletions(-)
create mode 100644 docs/nbdkit-filter.pod
create mode 100644 filters/Makefile.am
create mode 100644 filters/delay/Makefile.am
create mode 100644 filters/delay/delay.c
create mode 100644 filters/delay/nbdkit-delay-filter.pod
create mode 100644 filters/offset/Makefile.am
create mode 100644 filters/offset/nbdkit-offset-filter.pod
create mode 100644 filters/offset/offset.c
create mode 100644 include/nbdkit-filter.h
create mode 100644 src/filters.c
create mode 100644 src/locks.c
--
2.14.3
6 years, 9 months
[nbdkit] A short note about filters in nbdkit
by Richard W.M. Jones
I've added support for filters to the nbdkit upstream repository
(version >= 1.1.27). Filters allow you to modify plugins. A simple
example would be selecting a partition from a plugin:
nbdkit --filter=partition file file=disk.img partition=2
^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^
which would serve the second partition only from disk.img. For more
information see the nbdkit(1) and nbdkit-filter(3) man pages.
For nbdkit plugins written in C we offer a long term stable ABI, where
you can write a plugin (in C) now and have it work on all future
versions of nbdkit. This means plugins can be distributed separately
from nbdkit.
However for nbdkit filters we haven't decided what, if any, ABI
guarantees we will offer. The reason for this is simple: filters work
by filtering plugin methods, but if we add new plugin methods then
existing filters won't "see" the new methods. In some common cases
this could even cause data corruption, for example if a filter is
modifying file offsets like the partition filter above, then the new
methods would write to the wrong part of the file.
So the TL;DR is, for now, if you write a filter it is far better to
submit it upstream so that we can keep it up to date.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
6 years, 9 months
[PATCH nbdkit filters-v3 0/7] Introduce filters.
by Richard W.M. Jones
This is still tentative and needs a lot of work, but:
- partition filter works, supporting MBR & GPT
- prepare and finalize methods fixed
- open method can now be changed (allowing readonly flag to be modified)
- thread_model can be limited
I believe I made most of the changes which were previously suggested
in email. I think the only one I didn't was preventing inclusion of
both <nbdkit-plugin.h> and <nbdkit-filter.h> in the same file.
Rich.
6 years, 9 months
[PATCH nbdkit filters-v2 0/5] Introduce filters.
by Richard W.M. Jones
Rebased filters patch. Requires current git master + the locks /
thread model fix
(https://www.redhat.com/archives/libguestfs/2018-January/msg00128.html)
So a few changes here since last time:
The "introduce filters" and "implement filters" patches are
squashed together.
I introduced a concept of .prepare and .finalize. These run before
and after the data serving phase respectively. They're not a lot of
use for plugins because you can just use .open and .close instead, and
therefore I didn't expose them to plugins (only internally in the
backend). However for filters they make more sense since they give
you a place to prepare and finalize. They're necessary also for
filters because filter.open could not call the next open method, save
the (plugin) handle, then construct an nbdkit_next_ops object, and
call pread. It _has_ to be split into two operations.
This patch series is incomplete in at least three ways:
- We still need a way for filter.open to call down to the next open
function, so it can adjust the readonly flag (for copyonwrite
filter, TBD).
- Partition filter is obviously not complete.
- Need to write a copyonwrite filter.
Rich.
6 years, 9 months
[PATCH] launch: direct: Use old-style file= and format= parameters when not disabling locking (RHBZ#1503497).
by Richard W.M. Jones
Ancient qemu 1.5 (in RHEL 7) does not understand the
file.file.filename= and file.driver= parameters. Go back to using the
old-style file= and format= parameters when we're not trying to set
the file.backing.file.locking=off parameter.
Fixes commit 9fe592808ccfd9ed184b88ca9c6cad2e1798dee3.
Thanks: Yongkui Guo, Václav Kadlčík.
---
lib/launch-direct.c | 31 +++++++++++++++++++++----------
1 file changed, 21 insertions(+), 10 deletions(-)
diff --git a/lib/launch-direct.c b/lib/launch-direct.c
index aa4139359..47e8f37de 100644
--- a/lib/launch-direct.c
+++ b/lib/launch-direct.c
@@ -255,17 +255,28 @@ add_drive_standard_params (guestfs_h *g, struct backend_direct_data *data,
}
else {
/* Writable qcow2 overlay on top of read-only drive. */
- append_list_format ("file.file.filename=%s", drv->overlay);
- append_list ("file.driver=qcow2");
- append_list ("cache=unsafe");
- if (drv->disk_label)
- append_list_format ("serial=%s", drv->disk_label);
-
- /* Add the file-specific locking option only for files, as qemu
- * won't accept options unknown to the block driver in use.
- */
- if (data->qemu_mandatory_locking && drv->src.protocol == drive_protocol_file)
+ if (data->qemu_mandatory_locking &&
+ /* Add the file-specific locking option only for files, as
+ * qemu won't accept options unknown to the block driver in
+ * use.
+ */
+ drv->src.protocol == drive_protocol_file) {
+ append_list_format ("file.file.filename=%s", drv->overlay);
+ append_list ("file.driver=qcow2");
append_list ("file.backing.file.locking=off");
+ }
+ else {
+ /* Ancient qemu (esp. qemu 1.5 in RHEL 7) didn't understand the
+ * file.file.filename= parameter, so use the safer old-style
+ * form of parameters unless we actually want to specify the
+ * locking flag above.
+ */
+ append_list_format ("file=%s", drv->overlay);
+ append_list ("format=qcow2");
+ }
+ append_list ("cache=unsafe");
+ if (drv->disk_label)
+ append_list_format ("serial=%s", drv->disk_label);
}
append_list_format ("id=hd%zu", i);
--
2.13.2
6 years, 9 months