[nbdkit PATCH] nozero: Add notrim mode
by Eric Blake
It may be useful to test whether the client's use of
NBD_CMD_FLAG_NO_HOLE makes a difference; do this by adding a mode to
--filter=nozero to force a non-trimming zero write.
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
filters/nozero/nbdkit-nozero-filter.pod | 19 +++++++----
filters/nozero/nozero.c | 45 ++++++++++++++++++++-----
tests/test-nozero.sh | 27 +++++++++++++--
3 files changed, 74 insertions(+), 17 deletions(-)
diff --git a/filters/nozero/nbdkit-nozero-filter.pod b/filters/nozero/nbdkit-nozero-filter.pod
index 7e06570..8e694bb 100644
--- a/filters/nozero/nbdkit-nozero-filter.pod
+++ b/filters/nozero/nbdkit-nozero-filter.pod
@@ -18,14 +18,19 @@ testing client or server fallbacks.
=over 4
-=item B<zeromode=none|emulate>
+=item B<zeromode=none|emulate|notrim>
Optional, controls which mode the filter will use. Mode B<none>
-(default) means that zero support is not advertised to the client;
-mode B<emulate> means that zero support is emulated by the filter
-using the plugin's C<pwrite> callback, regardless of whether the
-plugin itself implemented the C<zero> callback with a more efficient
-way to write zeroes.
+(default) means that zero support is not advertised to the
+client. Mode B<emulate> means that zero support is emulated by the
+filter using the plugin's C<pwrite> callback, regardless of whether
+the plugin itself implemented the C<zero> callback with a more
+efficient way to write zeros. Since nbdkit E<ge> 1.13.4, mode
+B<notrim> means that zero requests are forwarded on to the plugin,
+except that the plugin will never see the NBDKIT_MAY_TRIM flag, to
+determine if the client permitting trimming during zero operations
+makes a difference (it is an error to request this mode if the plugin
+does not support the C<zero> callback).
=back
@@ -55,4 +60,4 @@ Eric Blake
=head1 COPYRIGHT
-Copyright (C) 2018 Red Hat Inc.
+Copyright (C) 2018-2019 Red Hat Inc.
diff --git a/filters/nozero/nozero.c b/filters/nozero/nozero.c
index 3ec6346..6e0ffa9 100644
--- a/filters/nozero/nozero.c
+++ b/filters/nozero/nozero.c
@@ -47,8 +47,12 @@
#define MAX_WRITE (64 * 1024 * 1024)
-static char buffer[MAX_WRITE];
-static bool emulate;
+static const char buffer[MAX_WRITE];
+static enum ZeroMode {
+ NONE,
+ EMULATE,
+ NOTRIM,
+} zeromode;
static int
nozero_config (nbdkit_next_config *next, void *nxdata,
@@ -56,7 +60,9 @@ nozero_config (nbdkit_next_config *next, void *nxdata,
{
if (strcmp (key, "zeromode") == 0) {
if (strcmp (value, "emulate") == 0)
- emulate = true;
+ zeromode = EMULATE;
+ else if (strcmp (value, "notrim") == 0)
+ zeromode = NOTRIM;
else if (strcmp (value, "none") != 0) {
nbdkit_error ("unknown zeromode '%s'", value);
return -1;
@@ -67,13 +73,31 @@ nozero_config (nbdkit_next_config *next, void *nxdata,
}
#define nozero_config_help \
- "zeromode=<MODE> Either 'none' (default) or 'emulate'.\n" \
+ "zeromode=<MODE> Either 'none' (default), 'emulate', or 'notrim'.\n" \
+
+/* Check that desired mode is supported by plugin. */
+static int
+nozero_prepare (struct nbdkit_next_ops *next_ops, void *nxdata, void *handle)
+{
+ int r;
+
+ if (zeromode == NOTRIM) {
+ r = next_ops->can_zero (nxdata);
+ if (r == -1)
+ return -1;
+ if (!r) {
+ nbdkit_error ("zeromode 'notrim' requires plugin zero support");
+ return -1;
+ }
+ }
+ return 0;
+}
/* Advertise desired WRITE_ZEROES mode. */
static int
nozero_can_zero (struct nbdkit_next_ops *next_ops, void *nxdata, void *handle)
{
- return emulate;
+ return zeromode != NONE;
}
static int
@@ -81,11 +105,15 @@ nozero_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
void *handle, uint32_t count, uint64_t offs, uint32_t flags,
int *err)
{
- assert (emulate);
+ assert (zeromode != NONE);
+ flags &= ~NBDKIT_FLAG_MAY_TRIM;
+
+ if (zeromode == NOTRIM)
+ return next_ops->zero (nxdata, count, offs, flags, err);
+
while (count) {
uint32_t size = MIN (count, MAX_WRITE);
- if (next_ops->pwrite (nxdata, buffer, size, offs,
- flags & ~NBDKIT_FLAG_MAY_TRIM, err) == -1)
+ if (next_ops->pwrite (nxdata, buffer, size, offs, flags, err) == -1)
return -1;
offs += size;
count -= size;
@@ -99,6 +127,7 @@ static struct nbdkit_filter filter = {
.version = PACKAGE_VERSION,
.config = nozero_config,
.config_help = nozero_config_help,
+ .prepare = nozero_prepare,
.can_zero = nozero_can_zero,
.zero = nozero_zero,
};
diff --git a/tests/test-nozero.sh b/tests/test-nozero.sh
index 1282586..fc22420 100755
--- a/tests/test-nozero.sh
+++ b/tests/test-nozero.sh
@@ -39,12 +39,14 @@ sock3=`mktemp -u`
sock4=`mktemp -u`
sock5a=`mktemp -u`
sock5b=`mktemp -u`
+sock6=`mktemp -u`
files="nozero1.img nozero1.log $sock1 nozero1.pid
nozero2.img nozero2.log $sock2 nozero2.pid
nozero3.img nozero3.log $sock3 nozero3.pid
nozero4.img nozero4.log $sock4 nozero4.pid
nozero5.img nozero5a.log nozero5b.log $sock5a $sock5b
- nozero5a.pid nozero5b.pid"
+ nozero5a.pid nozero5b.pid
+ nozero6.img nozero6.log $sock6 nozero6.pid"
rm -f $files
# Prep images, and check that qemu-io understands the actions we plan on
@@ -54,6 +56,7 @@ cp nozero1.img nozero2.img
cp nozero1.img nozero3.img
cp nozero1.img nozero4.img
cp nozero1.img nozero5.img
+cp nozero1.img nozero6.img
if ! qemu-io -f raw -d unmap -c 'w -z -u 0 1M' nozero1.img; then
echo "$0: missing or broken qemu-io"
rm nozero?.img
@@ -82,17 +85,20 @@ cleanup ()
cat nozero5a.log || :
echo "Log 5b file contents:"
cat nozero5b.log || :
+ echo "Log 6 file contents:"
+ cat nozero6.log || :
rm -f $files
}
cleanup_fn cleanup
-# Run four parallel nbdkit; to compare the logs and see what changes.
+# Run several parallel nbdkit; to compare the logs and see what changes.
# 1: unfiltered, to check that qemu-io sends ZERO request and plugin trims
# 2: log before filter with zeromode=none (default), to ensure no ZERO request
# 3: log before filter with zeromode=emulate, to ensure ZERO from client
# 4: log after filter with zeromode=emulate, to ensure no ZERO to plugin
# 5a/b: both sides of nbd plugin: even though server side does not advertise
# ZERO, the client side still exposes it, and just skips calling nbd's .zero
+# 6: log after filter with zeromode=notrim, to ensure plugin does not trim
start_nbdkit -P nozero1.pid -U $sock1 --filter=log \
file logfile=nozero1.log nozero1.img
start_nbdkit -P nozero2.pid -U $sock2 --filter=log --filter=nozero \
@@ -105,6 +111,8 @@ start_nbdkit -P nozero5a.pid -U $sock5a --filter=log --filter=nozero \
file logfile=nozero5a.log nozero5.img
start_nbdkit -P nozero5b.pid -U $sock5b --filter=log \
nbd logfile=nozero5b.log socket=$sock5a
+start_nbdkit -P nozero6.pid -U $sock6 --filter=nozero --filter=log \
+ file logfile=nozero6.log nozero6.img zeromode=notrim
# Perform the zero write.
qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock1"
@@ -112,6 +120,7 @@ qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock2"
qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock3"
qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock4"
qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock5b"
+qemu-io -f raw -c 'w -z -u 0 1M' "nbd+unix://?socket=$sock6"
# Check for expected ZERO vs. WRITE results
grep 'connection=1 Zero' nozero1.log
@@ -129,9 +138,23 @@ if grep 'connection=1 Zero' nozero5a.log; then
echo "nbdkit should have converted zero into write before nbd plugin"
exit 1
fi
+grep 'connection=1 Zero' nozero6.log
# Sanity check on contents - all 5 files should read identically
cmp nozero1.img nozero2.img
cmp nozero2.img nozero3.img
cmp nozero3.img nozero4.img
cmp nozero4.img nozero5.img
+cmp nozero5.img nozero6.img
+
+# Sanity check on sparseness; only image 1 should be sparse
+if test "$(stat -c %b nozero1.img)" = "$(stat -c %b nozero2.img)"; then
+ echo "nozero1.img was not trimmed"
+ exit 1
+fi
+for i in 3 4 5 6; do
+ if test "$(stat -c %b nozero2.img)" != "$(stat -c %b nozero$i.img)"; then
+ echo "nozero$i.img was trimmed by mistake"
+ exit 1
+ fi
+done
--
2.20.1
5 years, 6 months
[nbdkit PATCH] plugins: Use static buffer for plugin_zeroes
by Eric Blake
No need to calloc/free a buffer every time NBD_CMD_WRITE_ZEROES has to
fall back to a .pread call. Just reserve the maximum buffer up front
in our bss.
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
I noticed that buf was a candidate for CLEANUP_FREE, but then further
noticed that we can avoid the calloc/free altogether if we don't mind
the bss being 64M larger.
server/plugins.c | 10 ++--------
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/server/plugins.c b/server/plugins.c
index 947bb6d..acdfa1f 100644
--- a/server/plugins.c
+++ b/server/plugins.c
@@ -594,7 +594,7 @@ plugin_zero (struct backend *b, struct connection *conn,
uint32_t count, uint64_t offset, uint32_t flags, int *err)
{
struct backend_plugin *p = container_of (b, struct backend_plugin, backend);
- char *buf;
+ static const char buf[MAX_REQUEST_SIZE];
uint32_t limit;
int r = -1;
bool may_trim = flags & NBDKIT_FLAG_MAY_TRIM;
@@ -639,12 +639,7 @@ plugin_zero (struct backend *b, struct connection *conn,
assert (p->plugin.pwrite || p->plugin._pwrite_old);
flags &= ~NBDKIT_FLAG_MAY_TRIM;
threadlocal_set_error (0);
- limit = count < MAX_REQUEST_SIZE ? count : MAX_REQUEST_SIZE;
- buf = calloc (limit, 1);
- if (!buf) {
- *err = ENOMEM;
- return -1;
- }
+ limit = count < sizeof (buf) ? count : sizeof (buf);
while (count) {
r = plugin_pwrite (b, conn, buf, limit, offset, flags, err);
@@ -656,7 +651,6 @@ plugin_zero (struct backend *b, struct connection *conn,
}
*err = errno;
- free (buf);
errno = *err;
done:
--
2.20.1
5 years, 6 months
Welcome Hiroyuki Katsura as Google Summer of Code 2019 student
by Pino Toscano
Dear libguestfs developers and users,
it is my pleasure to welcome Hiroyuki Katsura as Google Summer of Code
2019 student for libguestfs!
He will work on adding Rust bindings, to allow using libguestfs in Rust
applications. You can read the project submission here:
https://summerofcode.withgoogle.com/projects/#6730007286644736
I'm the main mentor of this project, with Rich Jones, and
Martin Kletzander as co-mentors.
Hiroyuki: feel free to ask away about anything, from design questions to
doubts about the existing code, etc.
Welcome again, Hiroyuki!
--
Pino Toscano
5 years, 6 months
Re: [Libguestfs] [libguestfs/libguestfs] virt-tar-out (#37)
by Richard W.M. Jones
On Sun, May 12, 2019 at 04:11:08PM -0700, Jun Aruga wrote:
> I am using libguestfs tool on Fedora 30.
>
> libguestfs version: 1.40.2
> libguestfs-tools-c RPM package version: libguestfs-tools-c-1.40.2-4.fc30.x86_64
>
>
> I have a question about `virt-tar-out`.
>
> ```
> $ rpm -qf /usr/bin/virt-tar-out
> libguestfs-tools-c-1.40.2-4.fc30.x86_64
> ```
>
> Current master's `fish/virt-tar-out` https://github.com/libguestfs/libguestfs/blob/master/fish/virt-tar-out is almost same with my local `/usr/bin/virt-tar-out` except bash path.
>
> ```
> $ diff fish/virt-tar-out /usr/bin/virt-tar-out
> 1c1
> < #!/bin/bash -
> ---
> > #!/usr/bin/bash -
> ```
Something in Fedora itself rewrites this path, it's nothing to do with
libguestfs and in any case this should not cause a problem.
> ## case1: convert CentOS7 aarch64 iso image to the tar file.
>
> ```
> $ wget http://mirror.centos.org/altarch/7.6.1810/isos/aarch64/CentOS-7-aarch64-M...
> ```
>
> ```
> $ virt-tar-out -a CentOS-7-aarch64-Minimal-1810.iso / foo.tar
> virt-tar-out: no operating system was found on this disk
libguestfs cannot inspect installation ISOs which is why this fails.
> If using guestfish ‘-i’ option, remove this option and instead
> use the commands ‘run’ followed by ‘list-filesystems’.
> You can then mount filesystems you want by hand using the
> ‘mount’ or ‘mount-ro’ command.
>
> If using guestmount ‘-i’, remove this option and choose the
> filesystem(s) you want to see by manually adding ‘-m’ option(s).
> Use ‘virt-filesystems’ to see what filesystems are available.
>
> If using other virt tools, this disk image won’t work
> with these tools. Use the guestfish equivalent commands
> (see the virt tool manual page).
> ```
>
> This works when running `guestfish` directly like this.
>
> ```
> $ guestfish -a CentOS-7-aarch64-Minimal-1810.iso --ro <<EOF
> run
> list-filesystems
> EOF
> /dev/sda: iso9660
> ```
>
> ```
> $ guestfish --ro -a CentOS-7-aarch64-Minimal-1810.iso -m /dev/sda tar-out / foo.tar
>
> $ echo $?
> 0
>
> $ du -sh foo.tar
> 768M foo.tar
> ```
>
> ## case2: convert CentOS7 ppc64 iso image to the tar file.
>
> ```
> $ wget http://mirror.centos.org/altarch/7.6.1810/isos/ppc64/CentOS-7-ppc64-Minim...
> ```
>
> ```
> $ virt-tar-out -a CentOS-7-ppc64-Minimal-1810.iso foo.tar
> virt-tar-out: no operating system was found on this disk
>
> If using guestfish ‘-i’ option, remove this option and instead
> use the commands ‘run’ followed by ‘list-filesystems’.
> You can then mount filesystems you want by hand using the
> ‘mount’ or ‘mount-ro’ command.
>
> If using guestmount ‘-i’, remove this option and choose the
> filesystem(s) you want to see by manually adding ‘-m’ option(s).
> Use ‘virt-filesystems’ to see what filesystems are available.
>
> If using other virt tools, this disk image won’t work
> with these tools. Use the guestfish equivalent commands
> (see the virt tool manual page).
> ```
>
> This works when running `guestfish` directly like this.
>
> ```
> $ guestfish -a CentOS-7-ppc64-Minimal-1810.iso --ro <<EOF
> run
> list-filesystems
> EOF
> /dev/sda1: iso9660
> ```
>
> ```
> $ guestfish --ro -a CentOS-7-ppc64-Minimal-1810.iso -m /dev/sda1 tar-out / foo.tar
>
> $ echo $?
> 0
>
> $ du -sh foo.tar
> 877M foo.tar
> ```
>
> Why is `virt-tar-out` command not working?
> I should use `guestfish` command directly?
virt-tar-out is just a convenient, thin wrapper around guestfish, so
use guestfish if that works for you.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
5 years, 6 months
[PATCH 00/11] v2v: Change virt-v2v to use nbdkit for input in several modes.
by Richard W.M. Jones
This series (except the last one) changes virt-v2v to use nbdkit for
several input modes:
-i vmx -it vddk: No change in functionality, as this already uses
nbdkit-vddk-plugin, but the code is refactored for the other modes to
use.
-i libvirtxml: Use nbdkit-curl-plugin instead of qemu curl.
vCenter: Use nbdkit-curl-plugin instead of qemu curl.
xen: Use nbdkit-ssh-plugin instead of qemu ssh.
-i vmx -it ssh: Use nbdkit-ssh-plugin instead of qemu ssh.
I was hoping to post this patch series when it was longer and showed
some benefits. However those patches aren't ready quite yet, but
these ones are ready for review. However some of the future benefits
will be:
* using nbdkit-rate-plugin to bandwidth limit the input side
* better support for modern SSH encryption
* possibility of SSH password auth (instead of requiring ssh-agent)
I have tested this locally. Unfortunately none of the existing tests
cover any of these input cases.
The last patch reduces the minimum required version of nbdkit from
1.12 (not yet available) to 1.11, so this patch can be tested in
Fedora 30. It is obviously not meant to be applied.
Rich.
5 years, 6 months
Re: [Libguestfs] libguestfs Windows Build Support
by Richard W.M. Jones
On Wed, May 01, 2019 at 04:33:49PM +0300, Matan wrote:
> Hello Richard,
>
> I've recently encountered with libguestfs while looking for a library
> that provides parsing capabilities for VMDK formats, as part of a
> utility I'm working on. I've read the documentation and and few of the
> source code but couldn't find an answer for my question.
> Then, I've seen your that you are active on the mailing list, so I'm
> reaching out to you.
>
> Do you know of any way to build libguestfs on Windows? so that I'll
> have a compiled PE image that will be able to parse and mount VMDK
> files?
It used to build on Windows at some time in the very distant past, but
it doesn't now. I would suggest using virtualization similar to run
up a Linux environment.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
5 years, 6 months
