November 2017 - Libguestfs - libguestfs List Archives

[PATCH v13 0/3] virt-builder-repository
by Cédric Bosdonnat 22 Nov '17

22 Nov '17

Hey there, Here is an update of the series. The changes: * Incorporate Richard's comments. Left out the with_openfile one since that leads to a double close. * Change the ask option return type to string (removing the option) since if the use doesn't input anything we're using the default, and the default is now a mandatory parameter. * Make sure there are items in the lvs list before getting the first one. Cédric Bosdonnat (3): builder: change arch type to distinguish guesses builder: add a template parameter to get_index New tool: virt-builder-repository .gitignore | 4 + builder/Makefile.am | 87 ++++- builder/builder.ml | 6 +- builder/cache.ml | 6 +- builder/cache.mli | 6 +- builder/downloader.mli | 2 +- builder/index.ml | 9 +- builder/index.mli | 8 +- builder/index_parser.ml | 55 ++- builder/index_parser.mli | 5 +- builder/list_entries.ml | 6 +- builder/repository_main.ml | 595 ++++++++++++++++++++++++++++++++ builder/simplestreams_parser.ml | 2 +- builder/test-docs.sh | 2 + builder/test-virt-builder-repository.sh | 100 ++++++ builder/utils.ml | 4 + builder/utils.mli | 3 + builder/virt-builder-repository.pod | 213 ++++++++++++ builder/virt-builder.pod | 4 + fish/guestfish.pod | 1 + installcheck.sh.in | 1 + lib/guestfs.pod | 1 + 22 files changed, 1091 insertions(+), 29 deletions(-) create mode 100644 builder/repository_main.ml create mode 100755 builder/test-virt-builder-repository.sh create mode 100644 builder/virt-builder-repository.pod -- 2.15.0

3 5

[nbdkit PATCH v2 0/4] enable parallel nbd forwarding
by Eric Blake 21 Nov '17

21 Nov '17

With this, I am finally able to get the nbd plugin to do out-of-order responses to the client. Once this series goes in, we should be ready for Rich to cut a release. Eric Blake (4): nbd: Split reading into separate thread nbd: Protect writes with mutex nbd: Enable parallel handling tests: Test parallel nbd behavior plugins/nbd/nbd.c | 217 ++++++++++++++++++++++++++++++++++++--------- tests/Makefile.am | 1 + tests/test-parallel-nbd.sh | 81 +++++++++++++++++ 3 files changed, 255 insertions(+), 44 deletions(-) create mode 100755 tests/test-parallel-nbd.sh -- 2.13.6

2 9

[nbdkit PATCH] maint: Mention upstream NBD in README
by Eric Blake 21 Nov '17

21 Nov '17

It's worth having a link to the upstream NBD protocol as part of README. Fix a typo while at it. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- README | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README b/README index 7b5de90..7806cf5 100644 --- a/README +++ b/README @@ -11,7 +11,7 @@ The key features are: libraries or included in proprietary code. * Well-documented, simple plugin API with a stable ABI guarantee. - Let's you export "unconventional" block devices easily. + Lets you export "unconventional" block devices easily. * You can write plugins in C, Perl, Python, OCaml or Ruby. @@ -154,3 +154,4 @@ https://www.redhat.com/mailman/listinfo/libguestfs For further information, see: http://libguestfs.org/ +https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md -- 2.13.6

2 1

[nbdkit PATCH] nbd: Properly mop up stranded requests
by Eric Blake 21 Nov '17

21 Nov '17

Nothing in the second loop of nbd_reader() was ever setting quit; the only way the loop could end was via SEGV for dereferencing NULL or abort() when attempting to write to an invalid fd. Finish the implementation to properly mop up stranded in-flight requests. The testsuite did not exercise the case of quitting nbdkit while a transaction was in flight, but was relying on SIGTERM exiting the entire process before all threads were reaped; otherwise, I might have spotted this sooner. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- plugins/nbd/nbd.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/plugins/nbd/nbd.c b/plugins/nbd/nbd.c index a4a9639..df49a1d 100644 --- a/plugins/nbd/nbd.c +++ b/plugins/nbd/nbd.c @@ -384,15 +384,16 @@ nbd_reader (void *handle) } /* Clean up any stranded in-flight requests */ - done = false; r = ESHUTDOWN; - while (!done) { + while (1) { struct transaction *trans; nbd_lock (h); trans = h->trans; - h->trans = trans->next; + h->trans = trans ? trans->next : NULL; nbd_unlock (h); + if (!trans) + break; if (write (trans->u.fds[1], &r, sizeof r) != sizeof r) { nbdkit_error ("failed to write pipe: %m"); abort (); -- 2.13.6

2 1

[nbdkit PATCH v2 0/8] Support parallel transactions within single connection
by Eric Blake 21 Nov '17

21 Nov '17

I've posted some of these patches or ideas before; but now I'm confident enough with the series that it should be ready to push; at any rate, I can now run test-socket-activation in a tight loop without triggering any crashes or hangs. With this in place, I'm going back to work on making the nbd forwarder wort with the parallel thread model. Eric Blake (8): sockets: Use pipe-to-self to avoid hanging in poll cleanup: Add ACQUIRE_LOCK_FOR_CURRENT_SCOPE() connections: Add read/write lock over client I/O connections: Add thread-safe status indicator connections: Set up thread pool for handling client requests core: Add --threads option for supporting true parallel requests tests: Annotate Makefile.am conditionals tests: Test parallel behavior TODO | 7 - configure.ac | 11 +- docs/nbdkit.pod | 12 +- nbdkit.in | 2 +- plugins/file/file.c | 2 +- src/cleanup.c | 8 +- src/connections.c | 316 ++++++++++++++++++++++++++++++-------------- src/internal.h | 14 +- src/main.c | 39 +++++- src/plugins.c | 8 ++ src/sockets.c | 14 +- tests/Makefile.am | 55 +++++--- tests/test-parallel-file.sh | 71 ++++++++++ 13 files changed, 414 insertions(+), 145 deletions(-) create mode 100755 tests/test-parallel-file.sh -- 2.13.6

2 11

[PATCH v2] builder: planner: Don't add some impossible transitions.
by Richard W.M. Jones 21 Nov '17

21 Nov '17

Previous patch contained a typo. Changed output_format -> output_filename. Rich.

1 1

[PATCH] builder: planner: Don't add some impossible transitions.
by Richard W.M. Jones 21 Nov '17

21 Nov '17

Certain transitions where the input and output filename are the same are impossible, eg copying a file to itself. Don't add these. Reported-by: David Kaylor. --- builder/builder.ml | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/builder/builder.ml b/builder/builder.ml index a4c830e89..843106a86 100644 --- a/builder/builder.ml +++ b/builder/builder.ml @@ -413,6 +413,8 @@ let main () = let remove = List.remove_assoc in let ret = ref [] in + let infile = List.assoc `Filename itags in + (* The scheme for weights ranges from 0 = free to 100 = most expensive: * * 0 = free operations like renaming a file in the same directory @@ -433,8 +435,7 @@ let main () = * We could estimate weights better by looking at file sizes. *) let weight task otags = - let infile = List.assoc `Filename itags - and outfile = List.assoc `Filename otags in + let outfile = List.assoc `Filename otags in (* If infile/outfile don't exist, get the containing directory. *) let infile = @@ -474,14 +475,15 @@ let main () = * thing a copy does is to remove the template tag (since it's always * copied out of the cache directory). *) - tr `Copy ((`Filename, output_filename) :: remove `Template itags); + if infile <> output_format then + tr `Copy ((`Filename, output_filename) :: remove `Template itags); tr `Copy ((`Filename, tempfile) :: remove `Template itags); (* We can rename a file instead of copying, but don't rename the * cache copy! *) if is_not `Template then ( - if not output_is_block_dev then + if not output_is_block_dev && infile <> output_filename then tr `Move ((`Filename, output_filename) :: itags); tr `Move ((`Filename, tempfile) :: itags) ); @@ -490,7 +492,7 @@ let main () = (* If the input is XZ-compressed, then we can run xzcat, either * to the output file or to a temp file. *) - if not output_is_block_dev then + if not output_is_block_dev && infile <> output_filename then tr `Pxzcat ((`Filename, output_filename) :: remove `XZ (remove `Template itags)); tr `Pxzcat @@ -504,10 +506,11 @@ let main () = let old_size = Int64.of_string (List.assoc `Size itags) in let headroom = 256L *^ 1024L *^ 1024L in if output_size >= old_size +^ headroom then ( - tr `Virt_resize - ((`Size, Int64.to_string output_size) :: - (`Filename, output_filename) :: - (`Format, output_format) :: (remove `Template itags)); + if infile <> output_filename then + tr `Virt_resize + ((`Size, Int64.to_string output_size) :: + (`Filename, output_filename) :: + (`Format, output_format) :: (remove `Template itags)); tr `Virt_resize ((`Size, Int64.to_string output_size) :: (`Filename, tempfile) :: @@ -532,9 +535,10 @@ let main () = (* qemu-img convert is always possible, and quicker. It doesn't * resize, but it does change the format. *) - tr `Convert - ((`Filename, output_filename) :: (`Format, output_format) :: - (remove `Template itags)); + if infile <> output_filename then + tr `Convert + ((`Filename, output_filename) :: (`Format, output_format) :: + (remove `Template itags)); tr `Convert ((`Filename, tempfile) :: (`Format, output_format) :: (remove `Template itags)); -- 2.13.2

1 0

virt-builder resize error
by David Kaylor 21 Nov '17

21 Nov '17

Hi, I was trying out virt-builder and noticed that in some directories I receive an error when the image is resized. For example, if I run the following command from my home directory it works fine: virt-builder rhel-7.4 --size 10G --output test.img If I run the same command from /tmp or /vms (my default libvirt pool), I see the following in my verbose output: virt-resize '--verbose' '--format' 'raw' '--output-format' 'raw' '--expand' '/dev/sda3' '--unknown-filesystems' 'error' 'test.img' 'test.img' command line: virt-resize --verbose --format raw --output-format raw --expand /dev/sda3 --unknown-filesystems error test.img test.img virt-resize: error: you cannot use the same disk image for input and output Am I doing something wrong or is this possibly a bug? The host I am running virt-builder on is Fedora 27. Thanks, David

3 6

[PATCH 0/3] Small improvements and fixes to urandom.
by Richard W.M. Jones 21 Nov '17

21 Nov '17

Small improvements and fixes to urandom.

1 3

[nbdkit PATCH 0/2] Add nbd forwarder test coverage
by Eric Blake 21 Nov '17

21 Nov '17

To avoid bitrot, any new feature needs testsuite coverage ;) Still to come: once I get my work on parallel nbd finished, I will add a test-parallel-nbd.sh that closely mirrors what my other series added in test-parallel-file.sh. If desired, it might be a fun exercise to tweak test-nbd into using a for-loop of user-controlled depth for how deep you want to nest the forwarding tree, to see where the bottlenecks like in trying to improve nbdkit performance. Eric Blake (2): tests: Prepare for running multiple nbdkits in one test tests: Test nbd forwarder plugin .gitignore | 1 + tests/Makefile.am | 8 ++++ tests/test-nbd.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/test.c | 118 +++++++++++++++++++++++++++++++++++------------------ tests/test.h | 4 +- 5 files changed, 209 insertions(+), 41 deletions(-) create mode 100644 tests/test-nbd.c -- 2.13.6

2 3