[PATCH v13 0/3] virt-builder-repository
by Cédric Bosdonnat
Hey there,
Here is an update of the series. The changes:
* Incorporate Richard's comments. Left out the with_openfile one
since that leads to a double close.
* Change the ask option return type to string (removing the option)
since if the use doesn't input anything we're using the default,
and the default is now a mandatory parameter.
* Make sure there are items in the lvs list before getting the
first one.
Cédric Bosdonnat (3):
builder: change arch type to distinguish guesses
builder: add a template parameter to get_index
New tool: virt-builder-repository
.gitignore | 4 +
builder/Makefile.am | 87 ++++-
builder/builder.ml | 6 +-
builder/cache.ml | 6 +-
builder/cache.mli | 6 +-
builder/downloader.mli | 2 +-
builder/index.ml | 9 +-
builder/index.mli | 8 +-
builder/index_parser.ml | 55 ++-
builder/index_parser.mli | 5 +-
builder/list_entries.ml | 6 +-
builder/repository_main.ml | 595 ++++++++++++++++++++++++++++++++
builder/simplestreams_parser.ml | 2 +-
builder/test-docs.sh | 2 +
builder/test-virt-builder-repository.sh | 100 ++++++
builder/utils.ml | 4 +
builder/utils.mli | 3 +
builder/virt-builder-repository.pod | 213 ++++++++++++
builder/virt-builder.pod | 4 +
fish/guestfish.pod | 1 +
installcheck.sh.in | 1 +
lib/guestfs.pod | 1 +
22 files changed, 1091 insertions(+), 29 deletions(-)
create mode 100644 builder/repository_main.ml
create mode 100755 builder/test-virt-builder-repository.sh
create mode 100644 builder/virt-builder-repository.pod
--
2.15.0
6 years, 6 months
[nbdkit PATCH v2 0/4] enable parallel nbd forwarding
by Eric Blake
With this, I am finally able to get the nbd plugin to do out-of-order
responses to the client. Once this series goes in, we should be
ready for Rich to cut a release.
Eric Blake (4):
nbd: Split reading into separate thread
nbd: Protect writes with mutex
nbd: Enable parallel handling
tests: Test parallel nbd behavior
plugins/nbd/nbd.c | 217 ++++++++++++++++++++++++++++++++++++---------
tests/Makefile.am | 1 +
tests/test-parallel-nbd.sh | 81 +++++++++++++++++
3 files changed, 255 insertions(+), 44 deletions(-)
create mode 100755 tests/test-parallel-nbd.sh
--
2.13.6
6 years, 6 months
[nbdkit PATCH] maint: Mention upstream NBD in README
by Eric Blake
It's worth having a link to the upstream NBD protocol as part
of README.
Fix a typo while at it.
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
README | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/README b/README
index 7b5de90..7806cf5 100644
--- a/README
+++ b/README
@@ -11,7 +11,7 @@ The key features are:
libraries or included in proprietary code.
* Well-documented, simple plugin API with a stable ABI guarantee.
- Let's you export "unconventional" block devices easily.
+ Lets you export "unconventional" block devices easily.
* You can write plugins in C, Perl, Python, OCaml or Ruby.
@@ -154,3 +154,4 @@ https://www.redhat.com/mailman/listinfo/libguestfs
For further information, see:
http://libguestfs.org/
+https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md
--
2.13.6
6 years, 6 months
[nbdkit PATCH] nbd: Properly mop up stranded requests
by Eric Blake
Nothing in the second loop of nbd_reader() was ever setting
quit; the only way the loop could end was via SEGV for
dereferencing NULL or abort() when attempting to write to
an invalid fd. Finish the implementation to properly mop
up stranded in-flight requests.
The testsuite did not exercise the case of quitting
nbdkit while a transaction was in flight, but was relying
on SIGTERM exiting the entire process before all threads
were reaped; otherwise, I might have spotted this sooner.
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
plugins/nbd/nbd.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/plugins/nbd/nbd.c b/plugins/nbd/nbd.c
index a4a9639..df49a1d 100644
--- a/plugins/nbd/nbd.c
+++ b/plugins/nbd/nbd.c
@@ -384,15 +384,16 @@ nbd_reader (void *handle)
}
/* Clean up any stranded in-flight requests */
- done = false;
r = ESHUTDOWN;
- while (!done) {
+ while (1) {
struct transaction *trans;
nbd_lock (h);
trans = h->trans;
- h->trans = trans->next;
+ h->trans = trans ? trans->next : NULL;
nbd_unlock (h);
+ if (!trans)
+ break;
if (write (trans->u.fds[1], &r, sizeof r) != sizeof r) {
nbdkit_error ("failed to write pipe: %m");
abort ();
--
2.13.6
6 years, 6 months
[nbdkit PATCH v2 0/8] Support parallel transactions within single connection
by Eric Blake
I've posted some of these patches or ideas before; but now I'm
confident enough with the series that it should be ready to push;
at any rate, I can now run test-socket-activation in a tight loop
without triggering any crashes or hangs.
With this in place, I'm going back to work on making the nbd
forwarder wort with the parallel thread model.
Eric Blake (8):
sockets: Use pipe-to-self to avoid hanging in poll
cleanup: Add ACQUIRE_LOCK_FOR_CURRENT_SCOPE()
connections: Add read/write lock over client I/O
connections: Add thread-safe status indicator
connections: Set up thread pool for handling client requests
core: Add --threads option for supporting true parallel requests
tests: Annotate Makefile.am conditionals
tests: Test parallel behavior
TODO | 7 -
configure.ac | 11 +-
docs/nbdkit.pod | 12 +-
nbdkit.in | 2 +-
plugins/file/file.c | 2 +-
src/cleanup.c | 8 +-
src/connections.c | 316 ++++++++++++++++++++++++++++++--------------
src/internal.h | 14 +-
src/main.c | 39 +++++-
src/plugins.c | 8 ++
src/sockets.c | 14 +-
tests/Makefile.am | 55 +++++---
tests/test-parallel-file.sh | 71 ++++++++++
13 files changed, 414 insertions(+), 145 deletions(-)
create mode 100755 tests/test-parallel-file.sh
--
2.13.6
6 years, 6 months
[PATCH] builder: planner: Don't add some impossible transitions.
by Richard W.M. Jones
Certain transitions where the input and output filename are the same
are impossible, eg copying a file to itself. Don't add these.
Reported-by: David Kaylor.
---
builder/builder.ml | 28 ++++++++++++++++------------
1 file changed, 16 insertions(+), 12 deletions(-)
diff --git a/builder/builder.ml b/builder/builder.ml
index a4c830e89..843106a86 100644
--- a/builder/builder.ml
+++ b/builder/builder.ml
@@ -413,6 +413,8 @@ let main () =
let remove = List.remove_assoc in
let ret = ref [] in
+ let infile = List.assoc `Filename itags in
+
(* The scheme for weights ranges from 0 = free to 100 = most expensive:
*
* 0 = free operations like renaming a file in the same directory
@@ -433,8 +435,7 @@ let main () =
* We could estimate weights better by looking at file sizes.
*)
let weight task otags =
- let infile = List.assoc `Filename itags
- and outfile = List.assoc `Filename otags in
+ let outfile = List.assoc `Filename otags in
(* If infile/outfile don't exist, get the containing directory. *)
let infile =
@@ -474,14 +475,15 @@ let main () =
* thing a copy does is to remove the template tag (since it's always
* copied out of the cache directory).
*)
- tr `Copy ((`Filename, output_filename) :: remove `Template itags);
+ if infile <> output_format then
+ tr `Copy ((`Filename, output_filename) :: remove `Template itags);
tr `Copy ((`Filename, tempfile) :: remove `Template itags);
(* We can rename a file instead of copying, but don't rename the
* cache copy!
*)
if is_not `Template then (
- if not output_is_block_dev then
+ if not output_is_block_dev && infile <> output_filename then
tr `Move ((`Filename, output_filename) :: itags);
tr `Move ((`Filename, tempfile) :: itags)
);
@@ -490,7 +492,7 @@ let main () =
(* If the input is XZ-compressed, then we can run xzcat, either
* to the output file or to a temp file.
*)
- if not output_is_block_dev then
+ if not output_is_block_dev && infile <> output_filename then
tr `Pxzcat
((`Filename, output_filename) :: remove `XZ (remove `Template itags));
tr `Pxzcat
@@ -504,10 +506,11 @@ let main () =
let old_size = Int64.of_string (List.assoc `Size itags) in
let headroom = 256L *^ 1024L *^ 1024L in
if output_size >= old_size +^ headroom then (
- tr `Virt_resize
- ((`Size, Int64.to_string output_size) ::
- (`Filename, output_filename) ::
- (`Format, output_format) :: (remove `Template itags));
+ if infile <> output_filename then
+ tr `Virt_resize
+ ((`Size, Int64.to_string output_size) ::
+ (`Filename, output_filename) ::
+ (`Format, output_format) :: (remove `Template itags));
tr `Virt_resize
((`Size, Int64.to_string output_size) ::
(`Filename, tempfile) ::
@@ -532,9 +535,10 @@ let main () =
(* qemu-img convert is always possible, and quicker. It doesn't
* resize, but it does change the format.
*)
- tr `Convert
- ((`Filename, output_filename) :: (`Format, output_format) ::
- (remove `Template itags));
+ if infile <> output_filename then
+ tr `Convert
+ ((`Filename, output_filename) :: (`Format, output_format) ::
+ (remove `Template itags));
tr `Convert
((`Filename, tempfile) :: (`Format, output_format) ::
(remove `Template itags));
--
2.13.2
6 years, 6 months
virt-builder resize error
by David Kaylor
Hi,
I was trying out virt-builder and noticed that in some directories I
receive an error when the image is resized.
For example, if I run the following command from my home directory it works
fine:
virt-builder rhel-7.4 --size 10G --output test.img
If I run the same command from /tmp or /vms (my default libvirt pool), I
see the following in my verbose output:
virt-resize '--verbose' '--format' 'raw' '--output-format' 'raw' '--expand'
'/dev/sda3' '--unknown-filesystems' 'error' 'test.img' 'test.img'
command line: virt-resize --verbose --format raw --output-format raw
--expand /dev/sda3 --unknown-filesystems error test.img test.img
virt-resize: error: you cannot use the same disk image for input and output
Am I doing something wrong or is this possibly a bug? The host I am running
virt-builder on is Fedora 27.
Thanks,
David
6 years, 6 months
[nbdkit PATCH 0/2] Add nbd forwarder test coverage
by Eric Blake
To avoid bitrot, any new feature needs testsuite coverage ;)
Still to come: once I get my work on parallel nbd finished,
I will add a test-parallel-nbd.sh that closely mirrors what
my other series added in test-parallel-file.sh.
If desired, it might be a fun exercise to tweak test-nbd into
using a for-loop of user-controlled depth for how deep you
want to nest the forwarding tree, to see where the bottlenecks
like in trying to improve nbdkit performance.
Eric Blake (2):
tests: Prepare for running multiple nbdkits in one test
tests: Test nbd forwarder plugin
.gitignore | 1 +
tests/Makefile.am | 8 ++++
tests/test-nbd.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/test.c | 118 +++++++++++++++++++++++++++++++++++------------------
tests/test.h | 4 +-
5 files changed, 209 insertions(+), 41 deletions(-)
create mode 100644 tests/test-nbd.c
--
2.13.6
6 years, 6 months
