small question regarding ovirt
by Hetz Ben Hamo
Hi,
Small questoin: when running the P2v hard disk image, and I want to convert
a phyical machine to oVirt, to which machine do I connect when the GUI
starts? one of the nodes or the hosted engine? (i'm not talking about the
URL).
Thanks
5 years, 1 month
Plan for nbdkit 1.12
by Richard W.M. Jones
I've just uploaded the development version 1.11.12:
http://download.libguestfs.org/nbdkit/1.11-development/
Please try it out because it's hopefully close to the next stable
release, 1.12.
Below are some preliminary release notes. If there's anything else
that I've missed and should go into this version then let me know.
Rich.
======================================================================
New linuxdisk plugin lets you create complete ext2, ext3 or ext4
filesystems in a GPT partitioned disk image, from a local directory.
This can be attached to existing Linux VMs or used to create new ones
from scratch.
New ssh plugin which lets you turn any disk image on a remote Unix
server accessible by ssh into an NBD source.
New rate filter for bandwidth-limiting connections. Either single
connections or the whole server can be bandwidth-limited, and the
limit can be dynamically changed at runtime.
New readahead filter improves performance when reading sequentially
from plugins with a high request overhead like curl.
Plugins can now be written in the Rust programming language.
The old xz plugin has been removed. Use ‘nbdkit --filter=xz file file.xz’
as a replacement.
NBD_CMD_BLOCK_STATUS has been implemented allowing clients to query
which parts of the disk are allocated, holes or zeroes. Plugins and
filters have been extended to supply this information in many cases.
This required adding minimal support for NBD protocol Structured
Replies, NBD_CMD_FLAG_REQ_ONE and NBD_CMD_FLAG_DF. (Thanks to Eric
Blake for extensive help with this one.)
The line "All rights reserved." was removed from the license with
agreement from all contributors. Note this does not change the
license.
nbdkit_error errors are printed in red when writing to the terminal,
making them much easier to see.
nbdkit_parse_size rejects negative values, eg. ‘nbdkit memory size=-100’
(Nikolay Ivanets).
The curl plugin now supports cookies, making it broadly feature
equivalent to the qemu curl block driver.
The partitioning plugin can now create MBR logical partitions.
Previously it only supported 4 MBR primary partitions and if you
wanted more than that you had to use GPT.
The partition filter also supports MBR logical partitions.
The python plugin creates plugins which are compatible with Python >= 3.7.
Tests are now run under MALLOC_CHECK=1 and MALLOC_PERTURB_=<random>.
This causes some tests to crash (silently) because of a shutdown race
which needs to be fixed.
Test shell scripts now have a generic ‘requires’ function to skip
tests if features are not present.
C99 type ‘bool’ is now used in most places where appropriate.
The protocol and connections code in the server has been split up
because the single file had grown very large.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
5 years, 1 month
[PATCH v2 0/3] v2v: add -o json output mode
by Pino Toscano
This series adds a new output mode for virt-v2v, called -o json.
It produces local files, just like -o local, although the metadata
produced is a JSON file with data that v2v collected in the conversion
process. This can be useful for converting to unsupported destinations,
still based on QEMU/KVM.
In addition to a simple different metadata, it offers a way to relocate
the disks, with %{...}-like variables (only 3 added ATM, more can be
added) to change their paths depending on data of the guest/disks.
Changes from v1:
- remove extra code for optional named parameters in pcre_tests.ml
- allow %-escaping of variables in Var_expander
Pino Toscano (3):
common/mlpcre: add offset flag for PCRE.matches
v2v: add Var_expander
v2v: add -o json output mode
.gitignore | 1 +
common/mlpcre/PCRE.ml | 2 +-
common/mlpcre/PCRE.mli | 5 +-
common/mlpcre/pcre-c.c | 16 +-
common/mlpcre/pcre_tests.ml | 11 +-
v2v/Makefile.am | 36 +++-
v2v/cmdline.ml | 29 +++
v2v/create_json.ml | 348 ++++++++++++++++++++++++++++++++++
v2v/create_json.mli | 29 +++
v2v/dummy.c | 2 +
v2v/output_json.ml | 116 ++++++++++++
v2v/output_json.mli | 31 +++
v2v/var_expander.ml | 72 +++++++
v2v/var_expander.mli | 82 ++++++++
v2v/var_expander_tests.ml | 113 +++++++++++
v2v/virt-v2v-output-local.pod | 55 ++++++
v2v/virt-v2v.pod | 15 +-
17 files changed, 951 insertions(+), 12 deletions(-)
create mode 100644 v2v/create_json.ml
create mode 100644 v2v/create_json.mli
create mode 100644 v2v/dummy.c
create mode 100644 v2v/output_json.ml
create mode 100644 v2v/output_json.mli
create mode 100644 v2v/var_expander.ml
create mode 100644 v2v/var_expander.mli
create mode 100644 v2v/var_expander_tests.ml
--
2.20.1
5 years, 1 month
[PATCH nbdkit] Add readahead filter.
by Richard W.M. Jones
A suggested readahead filter. I've only lightly tested this, but it
seems to work fine with qemu-img convert. The commit needs proper
tests.
Rich.
5 years, 1 month
Readahead in the nbdkit curl plugin
by Richard W.M. Jones
I'm trying to replicate the features of the qemu curl plugin in
nbdkit's curl plugin, in order that we can use nbdkit in virt-v2v to
access VMware servers. I've implemented everything else so far [not
posted yet] except for readahead.
To my surprise actually, qemu's curl driver implements readahead
itself. I thought it was a curl feature. I'm not completely clear
_how_ it works in qemu, but it seems like it's maintaining an array of
outstanding AIO requests, searching those first to see if one contains
the read data, and extending the length of other read requests so
there is a higher chance they will prefetch data:
https://github.com/qemu/qemu/blob/230ce19814ecc6bff8edac3b5b86e7c82f422c6...
https://github.com/qemu/qemu/blob/230ce19814ecc6bff8edac3b5b86e7c82f422c6...
Oh and BTW in my testing we found that readahead was very important
for performance of virt-v2v! This is because vCenter is really slow
and we issue requests to vCenter serially, so avoiding long round
trips is vital. (Of course vCenter performance sucks and anyone with
any sense uses ‘virt-v2v -it ssh|vddk’, but vCenter unfortunately
remains the only zero-config way to convert guests with a pure free
software solution.)
So how could we implement something like this in nbdkit? I don't
particularly like the idea of extending the curl plugin so it's doing
caching as well (essentially what qemu does IIUC).
We already have a cache filter so I have two ideas:
(1) Modify nbdkit-cache-filter to add a readahead parameter.
This would work something like as follows: If the cache does not map
any data following a pread request (up to the size of the readahead),
then the pread request to the underlying plugin is extended, and the
data added to the cache (but not returned upwards). Prefetched data
is read from cache as usual.
Unfortunately this could slow down all pread requests, since there's
no easy way in a filter to return early with the requested data.
Background threads? We've avoided this in filters so far. One
problem is that they don't have access to the handle or "nextops"
structure.
(2) Add a new readahead filter which extents all pread requests
(unconditionally) but throws away the prefetched data.
You would need to use this in conjunction with a cache filter or other
caching layer. As Eric points out there is no way to enforce correct
layering here, we'd just have to trust the user.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
5 years, 1 month
[PATCH] OCaml tools: fix 3999 -> 3339 typo
by Pino Toscano
RFC 3339 is the actual RFC for date/time strings.
Typo found by Martin 'eagle eyes' Kletzander.
Fixes commit f79129b8dc92470e3a5597daf53c84038bd6859e.
---
common/mltools/tools_utils-c.c | 4 ++--
common/mltools/tools_utils.ml | 4 ++--
lib/guestfs.pod | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/common/mltools/tools_utils-c.c b/common/mltools/tools_utils-c.c
index b015dcace..3b80091c0 100644
--- a/common/mltools/tools_utils-c.c
+++ b/common/mltools/tools_utils-c.c
@@ -39,7 +39,7 @@
extern value guestfs_int_mllib_inspect_decrypt (value gv, value gpv, value keysv);
extern value guestfs_int_mllib_set_echo_keys (value unitv);
extern value guestfs_int_mllib_set_keys_from_stdin (value unitv);
-extern value guestfs_int_mllib_rfc3999_date_time_string (value unitv);
+extern value guestfs_int_mllib_rfc3339_date_time_string (value unitv);
/* Interface with the guestfish inspection and decryption code. */
int echo_keys = 0;
@@ -108,7 +108,7 @@ guestfs_int_mllib_set_keys_from_stdin (value unitv)
}
value
-guestfs_int_mllib_rfc3999_date_time_string (value unitv)
+guestfs_int_mllib_rfc3339_date_time_string (value unitv)
{
CAMLparam1 (unitv);
char buf[64];
diff --git a/common/mltools/tools_utils.ml b/common/mltools/tools_utils.ml
index de42df600..127180225 100644
--- a/common/mltools/tools_utils.ml
+++ b/common/mltools/tools_utils.ml
@@ -32,7 +32,7 @@ and key_store_key =
external c_inspect_decrypt : Guestfs.t -> int64 -> (string * key_store_key) list -> unit = "guestfs_int_mllib_inspect_decrypt"
external c_set_echo_keys : unit -> unit = "guestfs_int_mllib_set_echo_keys" "noalloc"
external c_set_keys_from_stdin : unit -> unit = "guestfs_int_mllib_set_keys_from_stdin" "noalloc"
-external c_rfc3999_date_time_string : unit -> string = "guestfs_int_mllib_rfc3999_date_time_string"
+external c_rfc3339_date_time_string : unit -> string = "guestfs_int_mllib_rfc3339_date_time_string"
type machine_readable_fn = {
pr : 'a. ('a, unit, string, unit) format4 -> 'a;
@@ -93,7 +93,7 @@ let log_as_json msgtype msg =
| Some { pr } ->
let json = [
"message", JSON.String msg;
- "timestamp", JSON.String (c_rfc3999_date_time_string ());
+ "timestamp", JSON.String (c_rfc3339_date_time_string ());
"type", JSON.String msgtype;
] in
pr "%s\n" (JSON.string_of_doc ~fmt:JSON.Compact json)
diff --git a/lib/guestfs.pod b/lib/guestfs.pod
index 3c1d635c5..af944ddb7 100644
--- a/lib/guestfs.pod
+++ b/lib/guestfs.pod
@@ -3295,7 +3295,7 @@ within a single line, below it is indented for readability):
C<type> can be: C<message> for progress messages, C<info> for
information messages, C<warning> for warning messages, and C<error>
for error message.
-C<timestamp> is the L<RFC 3999|https://www.ietf.org/rfc/rfc3339.txt>
+C<timestamp> is the L<RFC 3339|https://www.ietf.org/rfc/rfc3339.txt>
timestamp of the message.
In addition to that, a subset of these tools support an extra string
--
2.20.1
5 years, 1 month
guestfish Remote Images IPv6 Support
by Jonathan Wright
I have scoured the web and can't find anything on the topic: Is IPv6
supported for remote image targets?
For example:
guestfish --format=raw --ro -a
rbd://[fd00::cefc:1]:6789/images/CentOS-7-x86_64-GenericCloud-1901
Does not work citing the following:
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: create: flags = 0, handle = 0x5560231bdfb0, program = guestfish
libguestfs: trace: set_pgroup true
libguestfs: trace: set_pgroup = 0
libguestfs: trace: add_drive "images/CentOS-7-x86_64-GenericCloud-1901"
"readonly:true" "format:raw" "protocol:rbd" "server:tcp:[fd00::cefc:1]:6789"
libguestfs: creating COW overlay to protect original drive content
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: trace: disk_create "/tmp/libguestfs3pGMi6/overlay1.qcow2"
"qcow2" -1
"backingfile:rbd:images/CentOS-7-x86_64-GenericCloud-1901:mon_host=[fd00::cefc:1]\:6789:auth_supported=none"
"backingformat:raw"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o
backing_file=rbd:images/CentOS-7-x86_64-GenericCloud-1901:mon_host=[fd00::cefc:1]\:6789:auth_supported=none,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfs3pGMi6/overlay1.qcow2
qemu-img: /tmp/libguestfs3pGMi6/overlay1.qcow2: invalid conf option
:cefc:1]:6789:auth_supported: No such file or directory
Could not open backing image to determine size.
libguestfs: error: qemu-img: /tmp/libguestfs3pGMi6/overlay1.qcow2:
qemu-img exited with error status 1, see debug messages above
libguestfs: trace: disk_create = -1 (error)
libguestfs: trace: add_drive = -1 (error)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x5560231bdfb0 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfs3pGMi6
Other variations of the IPv6 address fail in a similar fashion, some
even failing with the URI parser. It seems as though everything after
the first : in the address is getting parsed as args.
Changing this IPv6 address over to a hostname allows the command to work.
Unfortunately I'm working within the confines of OpenStack and some of
it's requests here and my Ceph cluster is v6-only so I can't easily just
have it use hostnames to solve the issue.
--
Jonathan Wright
KnownHost, LLC
https://www.knownhost.com
5 years, 1 month
[nbdkit PATCH] protocol: Trivially implement NBD_CMD_FLAG_DF
by Eric Blake
The DF flag is only available to clients that negotiated structured
replies, and even then, the spec does not require that it be
implemented. However, since our current implementation can't fragment
NBD_CMD_READ replies, we trivially implement the flag (by ignoring
it); we don't even have to pass it on to the plugins.
Enhance some documentation about sparse reads while at it (when we
finally do allow plugins to implement sparse reads, we'll have to pass
on the DF flag, as well as perform reassembly back into one reply
ourselves if the plugin ignored the flag).
tests/test-eflags.sh can't really test this, as it requires the
negotiation of structured replies (which in turn requires newstyle,
not oldstyle...)
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
docs/nbdkit-protocol.pod | 13 +++++++++++++
common/protocol/protocol.h | 2 ++
server/protocol-handshake.c | 3 +++
server/protocol.c | 16 +++++++++++++++-
TODO | 10 +++++++---
5 files changed, 40 insertions(+), 4 deletions(-)
diff --git a/docs/nbdkit-protocol.pod b/docs/nbdkit-protocol.pod
index a9a3390..f706cfd 100644
--- a/docs/nbdkit-protocol.pod
+++ b/docs/nbdkit-protocol.pod
@@ -139,6 +139,19 @@ Supported in nbdkit E<ge> 1.11.10.
Only C<base:allocation> (ie. querying which parts of an image are
sparse) is supported.
+Sparse reads (using C<NBD_REPLY_TYPE_OFFSET_HOLE> are not directly
+supported, but a client can use block status to infer which portions
+of the export do not need to be read.
+
+=item C<NBD_FLAG_DF>
+
+Supported in nbdkit E<ge> 1.11.11.
+
+This protocol extension allows a client to force an all-or-none read
+when structured replies are in effect. However, the flag is a no-op
+until we extend the plugin API to allow a fragmented read in the first
+place.
+
=item Resize Extension
I<Not supported>.
diff --git a/common/protocol/protocol.h b/common/protocol/protocol.h
index a7de2f0..4334be4 100644
--- a/common/protocol/protocol.h
+++ b/common/protocol/protocol.h
@@ -94,6 +94,7 @@ extern const char *name_of_nbd_flag (int);
#define NBD_FLAG_ROTATIONAL (1 << 4)
#define NBD_FLAG_SEND_TRIM (1 << 5)
#define NBD_FLAG_SEND_WRITE_ZEROES (1 << 6)
+#define NBD_FLAG_SEND_DF (1 << 7)
#define NBD_FLAG_CAN_MULTI_CONN (1 << 8)
/* NBD options (new style handshake only). */
@@ -217,6 +218,7 @@ extern const char *name_of_nbd_cmd (int);
extern const char *name_of_nbd_cmd_flag (int);
#define NBD_CMD_FLAG_FUA (1<<0)
#define NBD_CMD_FLAG_NO_HOLE (1<<1)
+#define NBD_CMD_FLAG_DF (1<<2)
#define NBD_CMD_FLAG_REQ_ONE (1<<3)
/* Error codes (previously errno).
diff --git a/server/protocol-handshake.c b/server/protocol-handshake.c
index 9653210..f0e5a2c 100644
--- a/server/protocol-handshake.c
+++ b/server/protocol-handshake.c
@@ -121,6 +121,9 @@ protocol_compute_eflags (struct connection *conn, uint16_t *flags)
if (fl)
conn->can_extents = true;
+ if (conn->structured_replies)
+ eflags |= NBD_FLAG_SEND_DF;
+
*flags = eflags;
return 0;
}
diff --git a/server/protocol.c b/server/protocol.c
index 383938f..d94cd19 100644
--- a/server/protocol.c
+++ b/server/protocol.c
@@ -110,7 +110,7 @@ validate_request (struct connection *conn,
/* Validate flags */
if (flags & ~(NBD_CMD_FLAG_FUA | NBD_CMD_FLAG_NO_HOLE |
- NBD_CMD_FLAG_REQ_ONE)) {
+ NBD_CMD_FLAG_DF | NBD_CMD_FLAG_REQ_ONE)) {
nbdkit_error ("invalid request: unknown flag (0x%x)", flags);
*error = EINVAL;
return false;
@@ -121,6 +121,20 @@ validate_request (struct connection *conn,
*error = EINVAL;
return false;
}
+ if (flags & NBD_CMD_FLAG_DF) {
+ if (cmd != NBD_CMD_READ) {
+ nbdkit_error ("invalid request: DF flag needs READ request");
+ *error = EINVAL;
+ return false;
+ }
+ if (!conn->structured_replies) {
+ nbdkit_error ("invalid request: "
+ "%s: structured replies was not negotiated",
+ name_of_nbd_cmd (cmd));
+ *error = EINVAL;
+ return false;
+ }
+ }
if ((flags & NBD_CMD_FLAG_REQ_ONE) &&
cmd != NBD_CMD_BLOCK_STATUS) {
nbdkit_error ("invalid request: REQ_ONE flag needs BLOCK_STATUS request");
diff --git a/TODO b/TODO
index 81d692b..2b944e9 100644
--- a/TODO
+++ b/TODO
@@ -24,8 +24,8 @@ General ideas for improvements
to inform nbdkit when the response is ready:
https://www.redhat.com/archives/libguestfs/2018-January/msg00149.html
-* More NBD protocol features. The only currently missing feature is
- online resize.
+* More NBD protocol features. The only currently missing features are
+ sparse reads and online resize.
* Add a callback to let plugins request minimum alignment for the
buffer to pread/pwrite; useful for a plugin utilizing O_DIRECT or
@@ -216,7 +216,11 @@ using ‘#define NBDKIT_API_VERSION <version>’.
* pread could be changed to allow it to support Structured Replies
(SRs). This could mean allowing it to return partial data, holes,
- zeroes, etc.
+ zeroes, etc. For a client that negotiates SR coupled with a plugin
+ that supports .extents, the v2 protocol would allow us to at least
+ synthesize NBD_REPLY_TYPE_OFFSET_HOLE for less network traffic, even
+ though the plugin will still have to fully populate the .pread
+ buffer; the v3 protocol should make sparse reads more direct.
* Parameters should be systematized so that they aren't just (key,
value) strings. nbdkit should know the possible keys for the plugin
--
2.20.1
5 years, 1 month
