guestfish Remote Images IPv6 Support
by Jonathan Wright
I have scoured the web and can't find anything on the topic: Is IPv6
supported for remote image targets?
For example:
guestfish --format=raw --ro -a
rbd://[fd00::cefc:1]:6789/images/CentOS-7-x86_64-GenericCloud-1901
Does not work citing the following:
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: create: flags = 0, handle = 0x5560231bdfb0, program = guestfish
libguestfs: trace: set_pgroup true
libguestfs: trace: set_pgroup = 0
libguestfs: trace: add_drive "images/CentOS-7-x86_64-GenericCloud-1901"
"readonly:true" "format:raw" "protocol:rbd" "server:tcp:[fd00::cefc:1]:6789"
libguestfs: creating COW overlay to protect original drive content
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: trace: disk_create "/tmp/libguestfs3pGMi6/overlay1.qcow2"
"qcow2" -1
"backingfile:rbd:images/CentOS-7-x86_64-GenericCloud-1901:mon_host=[fd00::cefc:1]\:6789:auth_supported=none"
"backingformat:raw"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o
backing_file=rbd:images/CentOS-7-x86_64-GenericCloud-1901:mon_host=[fd00::cefc:1]\:6789:auth_supported=none,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfs3pGMi6/overlay1.qcow2
qemu-img: /tmp/libguestfs3pGMi6/overlay1.qcow2: invalid conf option
:cefc:1]:6789:auth_supported: No such file or directory
Could not open backing image to determine size.
libguestfs: error: qemu-img: /tmp/libguestfs3pGMi6/overlay1.qcow2:
qemu-img exited with error status 1, see debug messages above
libguestfs: trace: disk_create = -1 (error)
libguestfs: trace: add_drive = -1 (error)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x5560231bdfb0 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfs3pGMi6
Other variations of the IPv6 address fail in a similar fashion, some
even failing with the URI parser. It seems as though everything after
the first : in the address is getting parsed as args.
Changing this IPv6 address over to a hostname allows the command to work.
Unfortunately I'm working within the confines of OpenStack and some of
it's requests here and my Ceph cluster is v6-only so I can't easily just
have it use hostnames to solve the issue.
--
Jonathan Wright
KnownHost, LLC
https://www.knownhost.com
5 years, 1 month
[nbdkit PATCH] protocol: Trivially implement NBD_CMD_FLAG_DF
by Eric Blake
The DF flag is only available to clients that negotiated structured
replies, and even then, the spec does not require that it be
implemented. However, since our current implementation can't fragment
NBD_CMD_READ replies, we trivially implement the flag (by ignoring
it); we don't even have to pass it on to the plugins.
Enhance some documentation about sparse reads while at it (when we
finally do allow plugins to implement sparse reads, we'll have to pass
on the DF flag, as well as perform reassembly back into one reply
ourselves if the plugin ignored the flag).
tests/test-eflags.sh can't really test this, as it requires the
negotiation of structured replies (which in turn requires newstyle,
not oldstyle...)
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
docs/nbdkit-protocol.pod | 13 +++++++++++++
common/protocol/protocol.h | 2 ++
server/protocol-handshake.c | 3 +++
server/protocol.c | 16 +++++++++++++++-
TODO | 10 +++++++---
5 files changed, 40 insertions(+), 4 deletions(-)
diff --git a/docs/nbdkit-protocol.pod b/docs/nbdkit-protocol.pod
index a9a3390..f706cfd 100644
--- a/docs/nbdkit-protocol.pod
+++ b/docs/nbdkit-protocol.pod
@@ -139,6 +139,19 @@ Supported in nbdkit E<ge> 1.11.10.
Only C<base:allocation> (ie. querying which parts of an image are
sparse) is supported.
+Sparse reads (using C<NBD_REPLY_TYPE_OFFSET_HOLE> are not directly
+supported, but a client can use block status to infer which portions
+of the export do not need to be read.
+
+=item C<NBD_FLAG_DF>
+
+Supported in nbdkit E<ge> 1.11.11.
+
+This protocol extension allows a client to force an all-or-none read
+when structured replies are in effect. However, the flag is a no-op
+until we extend the plugin API to allow a fragmented read in the first
+place.
+
=item Resize Extension
I<Not supported>.
diff --git a/common/protocol/protocol.h b/common/protocol/protocol.h
index a7de2f0..4334be4 100644
--- a/common/protocol/protocol.h
+++ b/common/protocol/protocol.h
@@ -94,6 +94,7 @@ extern const char *name_of_nbd_flag (int);
#define NBD_FLAG_ROTATIONAL (1 << 4)
#define NBD_FLAG_SEND_TRIM (1 << 5)
#define NBD_FLAG_SEND_WRITE_ZEROES (1 << 6)
+#define NBD_FLAG_SEND_DF (1 << 7)
#define NBD_FLAG_CAN_MULTI_CONN (1 << 8)
/* NBD options (new style handshake only). */
@@ -217,6 +218,7 @@ extern const char *name_of_nbd_cmd (int);
extern const char *name_of_nbd_cmd_flag (int);
#define NBD_CMD_FLAG_FUA (1<<0)
#define NBD_CMD_FLAG_NO_HOLE (1<<1)
+#define NBD_CMD_FLAG_DF (1<<2)
#define NBD_CMD_FLAG_REQ_ONE (1<<3)
/* Error codes (previously errno).
diff --git a/server/protocol-handshake.c b/server/protocol-handshake.c
index 9653210..f0e5a2c 100644
--- a/server/protocol-handshake.c
+++ b/server/protocol-handshake.c
@@ -121,6 +121,9 @@ protocol_compute_eflags (struct connection *conn, uint16_t *flags)
if (fl)
conn->can_extents = true;
+ if (conn->structured_replies)
+ eflags |= NBD_FLAG_SEND_DF;
+
*flags = eflags;
return 0;
}
diff --git a/server/protocol.c b/server/protocol.c
index 383938f..d94cd19 100644
--- a/server/protocol.c
+++ b/server/protocol.c
@@ -110,7 +110,7 @@ validate_request (struct connection *conn,
/* Validate flags */
if (flags & ~(NBD_CMD_FLAG_FUA | NBD_CMD_FLAG_NO_HOLE |
- NBD_CMD_FLAG_REQ_ONE)) {
+ NBD_CMD_FLAG_DF | NBD_CMD_FLAG_REQ_ONE)) {
nbdkit_error ("invalid request: unknown flag (0x%x)", flags);
*error = EINVAL;
return false;
@@ -121,6 +121,20 @@ validate_request (struct connection *conn,
*error = EINVAL;
return false;
}
+ if (flags & NBD_CMD_FLAG_DF) {
+ if (cmd != NBD_CMD_READ) {
+ nbdkit_error ("invalid request: DF flag needs READ request");
+ *error = EINVAL;
+ return false;
+ }
+ if (!conn->structured_replies) {
+ nbdkit_error ("invalid request: "
+ "%s: structured replies was not negotiated",
+ name_of_nbd_cmd (cmd));
+ *error = EINVAL;
+ return false;
+ }
+ }
if ((flags & NBD_CMD_FLAG_REQ_ONE) &&
cmd != NBD_CMD_BLOCK_STATUS) {
nbdkit_error ("invalid request: REQ_ONE flag needs BLOCK_STATUS request");
diff --git a/TODO b/TODO
index 81d692b..2b944e9 100644
--- a/TODO
+++ b/TODO
@@ -24,8 +24,8 @@ General ideas for improvements
to inform nbdkit when the response is ready:
https://www.redhat.com/archives/libguestfs/2018-January/msg00149.html
-* More NBD protocol features. The only currently missing feature is
- online resize.
+* More NBD protocol features. The only currently missing features are
+ sparse reads and online resize.
* Add a callback to let plugins request minimum alignment for the
buffer to pread/pwrite; useful for a plugin utilizing O_DIRECT or
@@ -216,7 +216,11 @@ using ‘#define NBDKIT_API_VERSION <version>’.
* pread could be changed to allow it to support Structured Replies
(SRs). This could mean allowing it to return partial data, holes,
- zeroes, etc.
+ zeroes, etc. For a client that negotiates SR coupled with a plugin
+ that supports .extents, the v2 protocol would allow us to at least
+ synthesize NBD_REPLY_TYPE_OFFSET_HOLE for less network traffic, even
+ though the plugin will still have to fully populate the .pread
+ buffer; the v3 protocol should make sparse reads more direct.
* Parameters should be systematized so that they aren't just (key,
value) strings. nbdkit should know the possible keys for the plugin
--
2.20.1
5 years, 1 month
Re: [Libguestfs] few things I found about virt-p2v
by Richard W.M. Jones
[Please keep replies on the list]
On Sat, Mar 30, 2019 at 02:24:36PM +0300, Hetz Ben Hamo wrote:
> > > I spent the last few hours playing with virt-p2v, and here are few things
> > > that I found, please tell me on which to submit a bug report:
> > >
> > > 1. I created a boot image (without any parameters) and tested it using
> > > virt-manager. With QXL driver, when it loads the Xorg, it shows .. a
> > blank
> > > screen (I tried it tens of times). With VirtIO it works.
> >
> > Unclear, you'd probably need to look at the logs from the virtual
> > machine to see what's going on.
> >
>
> I mean the screen is totally blank after it boots and tries to go to X. You
> cannot type or see anything. if you want, I can record a video and show it
> to you.
A video is not going to help. Press the key to get to the console
after boot ([Alt]+[Ctrl]+[F2] or something similar) and have a look at
the journalctl output.
> > > 3. There network to connect to - I think GUI wise, that it's not
> > > understandable that you need to click the word "default" and change
> > the
> > > name manually. Perhaps change the name from "default" to something
> > like
> > > "Type Network Name"?
> >
> > Where exactly is this in the UI?
> >
>
> In the right bottom side, where you see the nic MAC address and near it by
> default you see the word "Default".
I'm not exactly sure, but if it's the "Network interfaces" section of
the second page of the wizard then that is the network that we try to
connect to.
> > > 4. Looks like there is no way to type a pool name to export to (if I
> > > select libvirt) or am I missing something? (let's say I have a pool
> > called
> > > NAS-10G). In the -os field I'm trying to put the NAS-10G (as the
> > virt-v2v
> > > man page suggests), but this seems not work.
> >
> > Put the pool name into the storage (-os) field.
> >
>
> that's the problem. Once I remove the default "/var/tmp" and try to put my
> pool name and starts the conversion, it thinks that I'm using the Local
> mode (and I selected the libvirt mode) and when it finishes and it fails,
> the end of the log shows:
>
> virt-v2v: error: -o libvirt: output pool ‘NAS-10G’ is not a directory
> (type='dir'). See virt-v2v-output-local(1)
It could well be a bug, but take a look at the virt-p2v logs which are
saved under /tmp/virt-p2v-* on the conversion server. That will tell
you what exact command line was used.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
5 years, 1 month
few things I found about virt-p2v
by Hetz Ben Hamo
Hi,
I spent the last few hours playing with virt-p2v, and here are few things
that I found, please tell me on which to submit a bug report:
1. I created a boot image (without any parameters) and tested it using
virt-manager. With QXL driver, when it loads the Xorg, it shows .. a blank
screen (I tried it tens of times). With VirtIO it works.
2. RHV-Upload - I see that the functionality is there, but there is a
missing field for password. Since I don't have oVirt installed at the
moment, does it asks for the password from the user? could someone please
add a box for this if the user chooses RHV-Upload?
3. There network to connect to - I think GUI wise, that it's not
understandable that you need to click the word "default" and change the
name manually. Perhaps change the name from "default" to something like
"Type Network Name"?
4. Looks like there is no way to type a pool name to export to (if I
select libvirt) or am I missing something? (let's say I have a pool called
NAS-10G). In the -os field I'm trying to put the NAS-10G (as the virt-v2v
man page suggests), but this seems not work.
5. Is there a way to access the ncurses mode? (for some really old
machines who don't respect VESA).
Thanks,
Hetz
5 years, 1 month
[PATCH 0/3] RFC: v2v: add -o json output mode
by Pino Toscano
This series adds a new output mode for virt-v2v, called -o json.
It produces local files, just like -o local, although the metadata
produced is a JSON file with data that v2v collected in the conversion
process. This can be useful for converting to unsupported destinations,
still based on QEMU/KVM.
In addition to a simple different metadata, it offers a way to relocate
the disks, with %{...}-like variables (only 3 added ATM, more can be
added) to change their paths depending on data of the guest/disks.
Thanks,
Pino Toscano (3):
common/mlpcre: add offset flag for PCRE.matches
v2v: add Var_expander
v2v: add -o json output mode
.gitignore | 1 +
common/mlpcre/PCRE.ml | 2 +-
common/mlpcre/PCRE.mli | 5 +-
common/mlpcre/pcre-c.c | 16 +-
common/mlpcre/pcre_tests.ml | 15 +-
v2v/Makefile.am | 36 +++-
v2v/cmdline.ml | 29 +++
v2v/create_json.ml | 348 ++++++++++++++++++++++++++++++++++
v2v/create_json.mli | 29 +++
v2v/dummy.c | 2 +
v2v/output_json.ml | 116 ++++++++++++
v2v/output_json.mli | 31 +++
v2v/var_expander.ml | 69 +++++++
v2v/var_expander.mli | 82 ++++++++
v2v/var_expander_tests.ml | 103 ++++++++++
v2v/virt-v2v-output-local.pod | 50 +++++
v2v/virt-v2v.pod | 15 +-
17 files changed, 937 insertions(+), 12 deletions(-)
create mode 100644 v2v/create_json.ml
create mode 100644 v2v/create_json.mli
create mode 100644 v2v/dummy.c
create mode 100644 v2v/output_json.ml
create mode 100644 v2v/output_json.mli
create mode 100644 v2v/var_expander.ml
create mode 100644 v2v/var_expander.mli
create mode 100644 v2v/var_expander_tests.ml
--
2.20.1
5 years, 1 month
[PATCH v2 0/4] OCaml tools: output messages as JSON machine
by Pino Toscano
Enhance the output in machine parseable mode, by outputting all the
messages of OCaml tools as JSON to the machine parseable stream.
Related, although not strictly needed for this (and thus can be split
if requested), is the addition of the fd format for the machine
readable stream.
Changes from v1:
- use Obj.magic to convert int -> Unix.file_descr
- add tests
Pino Toscano (4):
common/mltools: move the code for machine readable up
common/mltools: make sure machine readable output is flushed
common/mltools: allow fd for machine readable output
OCaml tools: output messages into JSON for machine readable
.gitignore | 1 +
common/mltools/Makefile.am | 39 ++++++-
common/mltools/parse_tools_messages_test.py | 118 ++++++++++++++++++++
common/mltools/test-machine-readable.sh | 7 ++
common/mltools/test-tools-messages.sh | 28 +++++
common/mltools/tools_messages_tests.ml | 46 ++++++++
common/mltools/tools_utils-c.c | 51 +++++++++
common/mltools/tools_utils.ml | 89 ++++++++++-----
lib/guestfs.pod | 24 ++++
9 files changed, 371 insertions(+), 32 deletions(-)
create mode 100644 common/mltools/parse_tools_messages_test.py
create mode 100755 common/mltools/test-tools-messages.sh
create mode 100644 common/mltools/tools_messages_tests.ml
--
2.20.1
5 years, 1 month
nbdkit design: varying the rate limit parameter
by Richard W.M. Jones
Hi Eric,
We may have our first user (Tomas, CC'd) of the rate filter to limit
NBD connections. As you recall it lets you do commands like:
nbdkit --filter=rate memory size=64G rate=10M
to limit network bandwidth to 10 Mbps. However a twist is that he
needs to vary the parameter while nbdkit is running in response to a
UI, and we haven't really thought about that before.
In the grand and noble tradition of qemu we don't actually systematize
parameters. For the server they are just (key, value) strings. (It's
even possible for filters to modify plugin parameters, thankfully
nothing does that.)
The only filter we have which does anything similar is the error
filter, where the error-file parameter lets you enable and disable the
filter at runtime based on the presence of a file.
rm -f /tmp/inject
nbdkit --filter=error file disk.img error-rate=100% error-file=/tmp/inject
sleep 60; touch /tmp/inject
I think our options are:
* One-time hack in the rate filter.
* One-time hack in the rate filter, but make it compatible with a
possible future revision of parameter handling. eg: If we decided
that "rate:file" would in future mean "read ‘rate’ parameter from a
file" then we could add this as a simple parameter now but
systematize it in future.
* Systematize parameters. This would require a great deal of work and
likely another version of the API. (I pushed a note into the TODO
file for this.)
If we went all the way and allowed parameter values to change, would
this be triggered only by simple file changes, or would we want an
administration interface to the server? (a la QMP)
But I wonder if you have any better ideas?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
5 years, 1 month
[PATCH 0/4] OCaml tools: output messages as JSON machine
by Pino Toscano
Enhance the output in machine parseable mode, by outputting all the
messages of OCaml tools as JSON to the machine parseable stream.
Related, although not strictly needed for this (and thus can be split
if requested), is the addition of the fd format for the machine
readable stream.
Pino Toscano (4):
common/mltools: move the code for machine readable up
common/mltools: make sure machine readable output is flushed
common/mltools: allow fd for machine readable output
OCaml tools: output messages into JSON for machine readable
common/mltools/Makefile.am | 2 +-
common/mltools/tools_utils-c.c | 68 ++++++++++++++++++++++++++
common/mltools/tools_utils.ml | 88 ++++++++++++++++++++++------------
lib/guestfs.pod | 24 ++++++++++
4 files changed, 151 insertions(+), 31 deletions(-)
--
2.20.1
5 years, 1 month
[PATCH nbdkit v4 00/15] Implement Block Status.
by Richard W.M. Jones
I'm not sure exactly which version we're up to, but let's say it's
version 4.
I'm a lot happier with this version:
- all filters have been reviewed and changed where I think that's necessary
- can_extents is properly defined and implemented now
- NBD protocol is followed
- I believe it addresses all previous review points where possible
The "only" thing missing is it desperately needs proper tests, which
is the next thing on my to-do list. So that means the patch series
still isn't complete and ready for upstream, but it's certainly ready
for review.
Rich.
5 years, 1 month