June 2018 - Libguestfs - libguestfs List Archives

HELP
by Shashwat shagun 07 Jun '18

07 Jun '18

Hi, i cannot import libguestfs golang lib, go get package libguestfs.org/guestfs: unrecognized import path " libguestfs.org/guestfs" (https fetch: Get https://libguestfs.org/ guestfs?go-get=1: dial tcp [2001:41c9:1:422::215]:443: getsockopt: connection refused) Any ideas? has the url changed or something? -- Regards, Shashwat Shagun

1 0

[PATCH nbdkit] tests: xz: Use 16M block size when preparing disk for xz plugin test.
by Richard W.M. Jones 06 Jun '18

06 Jun '18

FYI; I'm going to push this because it's such an obvious fix. Rich.

1 1

[PATCH] tests: Provide full path to Unix domain sockets.
by Richard W.M. Jones 06 Jun '18

06 Jun '18

Maybe a bug in libvirt? https://www.redhat.com/archives/libvir-list/2018-June/msg00490.html --- tests/test-cache.sh | 3 ++- tests/test-cow.sh | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/test-cache.sh b/tests/test-cache.sh index e5e9e82..ac99ce2 100755 --- a/tests/test-cache.sh +++ b/tests/test-cache.sh @@ -32,6 +32,7 @@ # SUCH DAMAGE. set -e +set -x files="cache.img cache.sock cache.pid" rm -f $files @@ -69,7 +70,7 @@ cleanup () trap cleanup INT QUIT TERM EXIT ERR # Open the overlay and perform some operations. -guestfish --format=raw -a 'nbd://?socket=cache.sock' <<'EOF' +guestfish --format=raw -a "nbd://?socket=`pwd`/cache.sock" <<'EOF' run part-disk /dev/sda gpt mkfs ext4 /dev/sda1 diff --git a/tests/test-cow.sh b/tests/test-cow.sh index 443ba38..4b798c8 100755 --- a/tests/test-cow.sh +++ b/tests/test-cow.sh @@ -32,6 +32,7 @@ # SUCH DAMAGE. set -e +set -x files="cow-base.img cow-diff.qcow2 cow.sock cow.pid" rm -f $files @@ -70,7 +71,7 @@ cleanup () trap cleanup INT QUIT TERM EXIT ERR # Write some data into the overlay. -guestfish --format=raw -a 'nbd://?socket=cow.sock' -m /dev/sda1 <<EOF +guestfish --format=raw -a "nbd://?socket=`pwd`/cow.sock" -m /dev/sda1 <<EOF fill-dir / 10000 fill-pattern "abcde" 5M /large write /hello "hello, world" -- 2.16.2

2 2

[PATCH nbdkit] locks: Remove debugging messages about
by Richard W.M. Jones 06 Jun '18

06 Jun '18

The messages are not really useful to us, but they do bloat the debugging output of virt-v2v massively: nbdkit: python[1]: debug: acquire global request lock nbdkit: python[1]: debug: acquire per-connection request lock nbdkit: python[1]: debug: acquire unload prevention lock nbdkit: python[1]: debug: pwrite count=2097152 offset=4628414464 fua=0 nbdkit: python[1]: debug: release unload prevention lock nbdkit: python[1]: debug: release per-connection request lock nbdkit: python[1]: debug: release global request lock We probably need some kind of level of verbosity to deal with this sort of thing, but that's quite a large change. Rich.

2 2

[PATCH 0/3] v2v: Various refactorings.
by Richard W.M. Jones 06 Jun '18

06 Jun '18

Use -ip instead of --password-file, and various refactorings. It strikes me that we should probably deprecate and eventually remove virt-v2v-copy-to-local. With the introduction of the new SSH and VDDK transports, and with RHEL 5 Xen becoming more irrelevant, it's no longer needed. Rich.

2 4

FYI: CVE-2018-11806 Qemu: slirp: heap buffer overflow while reassembling fragmented datagrams
by Richard W.M. Jones 06 Jun '18

06 Jun '18

There's a heap overflow in qemu SLIRP which affects libguestfs, potentially allowing a malicious filesystem to take control of the confining qemu process and from there attack the host. It will affect libguestfs specifically when these two conditions are both met: - You're using the ‘direct’ backend. - Networking is enabled. The direct backend is the default upstream, but not in Fedora/RHEL/CentOS. It might also have been selected if you set the LIBGUESTFS_BACKEND=direct environment variable or called ‘guestfs_set_backend (g, "direct")’. Networking is enabled automatically by some tools (eg. virt-builder), or is enabled if your code called ‘guestfs_set_network (g, 1)’ (which is not the default). Note that the libvirt backend is _not_ affected, for two reasons: (1) It doesn't use SLIRP for networking (2) libvirt will use SELinux to confine the qemu process, so even if the appliance gains full control over qemu it is limited in what it can do. The solution in any case is to upgrade to a qemu containing the fix. Here's the upstream qemu patch: https://lists.gnu.org/archive/html/qemu-devel/2018-06/msg01012.html Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

1 0

[PATCH] v2v: Log full imageio response on failure.
by Richard W.M. Jones 05 Jun '18

05 Jun '18

Thanks: Nir Soffer --- v2v/rhv-upload-plugin.py | 66 ++++++++++++++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 27 deletions(-) diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py index 791c9e7d2..c3de7d555 100644 --- a/v2v/rhv-upload-plugin.py +++ b/v2v/rhv-upload-plugin.py @@ -228,6 +228,29 @@ def can_flush(h): def get_size(h): return params['disk_size'] +# Any unexpected HTTP response status from the server will end up +# calling this function which logs the full error, pauses the +# transfer, sets the failed state, and raises a RuntimeError +# exception. +def unexpected_response(h, r, msg): + # Setting the failed flag in the handle causes the disk to be + # cleaned up on close. + h['failed'] = True + h['transfer_service'].pause() + + status = r.status + reason = r.reason + body = r.read() + + # Log the full error if we're verbose. + debug("unexpected response from imageio server:") + debug(msg) + debug("%d: %s" % (status, reason)) + debug(body) + + # Only a short error is included in the exception. + raise RuntimeError("%s: %d: %s" % (msg, status, reason)) + # For documentation see: # https://github.com/oVirt/ovirt-imageio/blob/master/docs/random-io.md # For examples of working code to read/write from the server, see: @@ -248,16 +271,14 @@ def pread(h, count, offset): r = http.getresponse() # 206 = HTTP Partial Content. if r.status != 206: - h['transfer_service'].pause() - h['failed'] = True - raise RuntimeError("could not read sector (%d, %d): %d: %s" % - (offset, count, r.status, r.reason)) + unexpected_response(h, r, + "could not read sector offset %d size %d" % + (offset, count)) return r.read() def pwrite(h, buf, offset): http = h['http'] transfer = h['transfer'] - transfer_service = h['transfer_service'] count = len(buf) h['highestwrite'] = max(h['highestwrite'], offset+count) @@ -275,15 +296,13 @@ def pwrite(h, buf, offset): r = http.getresponse() if r.status != 200: - transfer_service.pause() - h['failed'] = True - raise RuntimeError("could not write sector (%d, %d): %d: %s" % - (offset, count, r.status, r.reason)) + unexpected_response(h, r, + "could not write sector offset %d size %d" % + (offset, count)) def zero(h, count, offset, may_trim): http = h['http'] transfer = h['transfer'] - transfer_service = h['transfer_service'] # Unlike the trim and flush calls, there is no 'can_zero' method # so nbdkit could call this even if the server doesn't support @@ -306,10 +325,9 @@ def zero(h, count, offset, may_trim): r = http.getresponse() if r.status != 200: - transfer_service.pause() - h['failed'] = True - raise RuntimeError("could not zero sector (%d, %d): %d: %s" % - (offset, count, r.status, r.reason)) + unexpected_response(h, r, + "could not zero sector offset %d size %d" % + (offset, count)) def emulate_zero(h, count, offset): # qemu-img convert starts by trying to zero/trim the whole device. @@ -334,15 +352,13 @@ def emulate_zero(h, count, offset): r = http.getresponse() if r.status != 200: - transfer_service.pause() - h['failed'] = True - raise RuntimeError("could not write zeroes (%d, %d): %d: %s" % - (offset, count, r.status, r.reason)) + unexpected_response(h, r, + "could not write zeroes offset %d size %d" % + (offset, count)) def trim(h, count, offset): http = h['http'] transfer = h['transfer'] - transfer_service = h['transfer_service'] # Construct the JSON request for trimming. buf = json.dumps({'op': "trim", @@ -358,15 +374,13 @@ def trim(h, count, offset): r = http.getresponse() if r.status != 200: - transfer_service.pause() - h['failed'] = True - raise RuntimeError("could not trim sector (%d, %d): %d: %s" % - (offset, count, r.status, r.reason)) + unexpected_response(h, r, + "could not trim sector offset %d size %d" % + (offset, count)) def flush(h): http = h['http'] transfer = h['transfer'] - transfer_service = h['transfer_service'] # Construct the JSON request for flushing. buf = json.dumps({'op': "flush"}).encode() @@ -379,9 +393,7 @@ def flush(h): r = http.getresponse() if r.status != 200: - transfer_service.pause() - h['failed'] = True - raise RuntimeError("could not flush: %d: %s" % (r.status, r.reason)) + unexpected_response(h, r, "could not flush") def delete_disk_on_failure(h): disk_service = h['disk_service'] -- 2.16.2

2 2

virt-builder download issues
by Alex Buchanan 01 Jun '18

01 Jun '18

I'm having trouble downloading templates (ubuntu-18.04, among others) using virt-builder. The downloads get to about 50% (or 13, or 62) and then stall. wget easily downloads the files quickly. I'd be happy with working around this by manually placing the images in the cache directory, but I can't seem to figure out the appropriate placement. virt-builder doesn't find the cached file. Please send help, Thanks!

2 2

[PATCH v2] daemon: inspect: better handling windows drive mapping.
by Mykola Ivanets 01 Jun '18

01 Jun '18

I saw several Windows disk images which contains strange registry entry for mapped drives: "\\DosDevices\\Y:"=hex(3):00,00,00,00,00,00,00,00,00,00,00,00 Which is decoded something like diskID = 0x0, partition starts at 0 bytes offset from the start of the disk. In addition to a Windows disk image, I have attached dummy disk and made xfs file system on a whole device without partitioning it. I mount xfs file system to a "/" and then mkdir and mount other found file systems inside (/fs1, /fs2 etc.). When we decode drive mappings we are looking for a disk with ID 0x0 (it is 4 bytes somewhere LBA0). It is appeared that dummy non-partitioned disk with xfs file system has zeros by offset where diskID is expected to be). So the disk is considered as a candidate to search for partition at offset 0. part-list command (and "parted" which is used under the hood) reports there is 1 partition on "dummy" disk which starts exactly at offset 0. And thus dummy device name and partition number are simply concatenated together and corresponding drive mapping is returned: Y => /dev/sdX1. But /dev/sdX1 is not existing block device. No matter either it is a bug in "parted" (or it works this way by-design), let's protect ourself from this situation: in addition we look for msdos partition table on a disk before making any further assumptions. --- daemon/inspect_fs_windows.ml | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/daemon/inspect_fs_windows.ml b/daemon/inspect_fs_windows.ml index 8b2aad8d3..af52dee2d 100644 --- a/daemon/inspect_fs_windows.ml +++ b/daemon/inspect_fs_windows.ml @@ -329,7 +329,7 @@ and get_drive_mappings h root data = String.is_prefix blob "DMIO:ID:" (* GPT *) then map_registry_disk_blob_gpt (Lazy.force partitions) blob else if String.length blob = 12 then - map_registry_disk_blob (Lazy.force devices) blob + map_registry_disk_blob_mbr (Lazy.force devices) blob else None ) @@ -356,13 +356,18 @@ and get_drive_mappings h root data = * The following function maps this blob to a libguestfs partition * name, if possible. *) -and map_registry_disk_blob devices blob = +and map_registry_disk_blob_mbr devices blob = try (* First 4 bytes are the disk ID. Search all devices to find the * disk with this disk ID. *) let diskid = String.sub blob 0 4 in - let device = List.find (fun dev -> pread dev 4 0x01b8 = diskid) devices in + let device = + List.find ( + fun dev -> + Parted.part_get_parttype dev = "msdos" && + pread dev 4 0x01b8 = diskid + ) devices in (* Next 8 bytes are the offset of the partition in bytes(!) given as * a 64 bit little endian number. Luckily it's easy to get the -- 2.17.0

2 1

[PATCH v8 0/6] daemon: list_filesystems: filter out block devices which cannot hold filesystem.
by Mykola Ivanets 01 Jun '18

01 Jun '18

v8: - Rebased on top of master. v7: - Addresses comments after v6 series review. v6: - Addresses comments after v5 series review. - Large commit is splitted to more granular commits for better code review. v5: - Addresses comments after v4 series review (part_get_mbr_part_type doesn't break original implementation in C). - Rebased on top of master and little bit refactored for readability. v4: - Rebased on top of master. v3: - Rebased on top of master. v2: - First implementation. v1: - Initial discussion. This patch series filters out block devices which cannot hold filesystems: - partitioned MD devices; - LDM partitions (only LDM volume can hold filesystem); - Windows Logical Disk Manager data partition; - Microsoft Reserved Partition; - Windows Snapshot Partition; - MBR extended partition. Mykola Ivanets (6): daemon: Changing the way that we detect if a device contains partitions. daemon: list-filesystems: Ignore partitioned MD devices. tests: list-filesystems command ignores partitioned MD devices. daemon: list-filesystems: Change the way we filter out LDM partitions. daemon: list-filesystems: Filter out Microsoft Reserved and Windows Snapshot partitions. daemon: list-filesystems: Filter out MBR extended partitions. daemon/listfs.ml | 132 ++++++++++++++---------- tests/md/Makefile.am | 3 +- tests/md/test-partitioned-md-devices.sh | 79 ++++++++++++++ 3 files changed, 157 insertions(+), 57 deletions(-) create mode 100755 tests/md/test-partitioned-md-devices.sh -- 2.17.0

2 7