nbdkit: Could not read L1 table when reading exported qcow2
Hello, I face a problem on the nbdkit server: Version-Release number of selected component (if applicable): nbdkit-v1.9.1 qemu-utils-2.12+dfsg-3+b1 How reproducible: 100% Steps: 1. Create a qcow2 file # qemu-img create disk 1G -f qcow2 2. Export the qcow2 file then try to read it via nbd url # nbdkit file file=disk -e raw -v nbdkit: debug: TLS disabled: could not load TLS certificates nbdkit: debug: registering plugin /usr/local/lib/nbdkit/plugins/nbdkit-file-plugin.so nbdkit: debug: registered plugin /usr/local/lib/nbdkit/plugins/nbdkit-file-plugin.so (name file) nbdkit: debug: file: load nbdkit: debug: file: config key=file, value=/tmp/disk nbdkit: debug: file: config_complete nbdkit: debug: bound to IP address <any>:10809 (2 socket(s)) nbdkit: debug: forked into background (new pid = 32524) 3. Try to read the file via nbd: # qemu-img info nbd://localhost/raw qemu-img: Could not open 'nbd://localhost/raw': Could not read L1 table: Invalid argument Logs from nbdkit: nbdkit: debug: file: open readonly=0 nbdkit: file[1]: debug: newstyle negotiation: flags: global 0x3 nbdkit: file[1]: debug: newstyle negotiation: client flags: 0x3 nbdkit: file[1]: debug: newstyle negotiation: NBD_OPT_GO: client requested export 'raw' (ignored) nbdkit: file[1]: debug: get_size nbdkit: file[1]: debug: can_write nbdkit: file[1]: debug: can_zero nbdkit: file[1]: debug: can_write nbdkit: file[1]: debug: can_trim nbdkit: file[1]: debug: can_fua nbdkit: file[1]: debug: can_flush nbdkit: file[1]: debug: is_rotational nbdkit: file[1]: debug: newstyle negotiation: flags: export 0x6d nbdkit: file[1]: debug: newstyle negotiation: NBD_OPT_GO: ignoring NBD_INFO_* request 3 nbdkit: file[1]: debug: handshake complete, processing requests with 16 threads nbdkit: debug: starting worker thread file.0 nbdkit: debug: starting worker thread file.1 nbdkit: debug: starting worker thread file.3 nbdkit: debug: starting worker thread file.2 nbdkit: debug: starting worker thread file.4 nbdkit: debug: starting worker thread file.5 nbdkit: debug: starting worker thread file.7 nbdkit: debug: starting worker thread file.6 nbdkit: debug: starting worker thread file.8 nbdkit: debug: starting worker thread file.9 nbdkit: debug: starting worker thread file.10 nbdkit: debug: starting worker thread file.11 nbdkit: debug: starting worker thread file.12 nbdkit: debug: starting worker thread file.13 nbdkit: debug: starting worker thread file.14 nbdkit: debug: starting worker thread file.15 nbdkit: file.1: debug: pread count=512 offset=0 nbdkit: file.0: debug: pread count=512 offset=0 nbdkit: file.3: debug: sending error reply: Invalid argument nbdkit: file.4: debug: client sent disconnect command, closing connection nbdkit: file.4: debug: exiting worker thread file.4 nbdkit: file.2: debug: exiting worker thread file.2 nbdkit: file.7: debug: exiting worker thread file.7 nbdkit: file.5: debug: exiting worker thread file.5 nbdkit: file.6: debug: exiting worker thread file.6 nbdkit: file.8: debug: exiting worker thread file.8 nbdkit: file.9: debug: exiting worker thread file.9 nbdkit: file.10: debug: exiting worker thread file.10 nbdkit: file.11: debug: exiting worker thread file.11 nbdkit: file.12: debug: exiting worker thread file.12 nbdkit: file.0: debug: exiting worker thread file.0 nbdkit: file.14: debug: exiting worker thread file.14 nbdkit: file.15: debug: exiting worker thread file.15 nbdkit: file.1: debug: exiting worker thread file.1 nbdkit: file.13: debug: exiting worker thread file.13 nbdkit: file.3: debug: exiting worker thread file.3 nbdkit: file[1]: debug: close Could you please check if it is a bug from nbdkit? Thanks -- Best regards, ----------------------------------- Han Han Quality Engineer Redhat. Email: hhan@redhat.com Phone: +861065339333
On Wed, Nov 21, 2018 at 11:33:33AM +0800, Han Han wrote:
Hello, I face a problem on the nbdkit server:
Version-Release number of selected component (if applicable): nbdkit-v1.9.1 qemu-utils-2.12+dfsg-3+b1
How reproducible: 100%
Steps: 1. Create a qcow2 file # qemu-img create disk 1G -f qcow2
This creates a 1GB virtual size qcow2 (real size is about 196K): $ ll disk -rw-rw-r--. 1 rjones rjones 196624 Nov 21 09:20 disk $ file disk disk: QEMU QCOW Image (v3), 1073741824 bytes
2. Export the qcow2 file then try to read it via nbd url # nbdkit file file=disk -e raw -v
This exports the disk using the export name "raw". However nbdkit doesn't know anything about qcow2, so this exports bytes of qcow2 data to the client. The virtual size of the disk will be about 196K, and it won't be resizable (so you cannot write to it). Note that the -e (export name) option is not used by nbdkit so you don't need it.
3. Try to read the file via nbd: # qemu-img info nbd://localhost/raw
What precisely happens here is that qemu-img sees a qcow2 file and tries to read beyond the end of the file: nbdkit: file.2: error: invalid request: offset and count are out of range: offset=196608 count=512 If you want to export a qcow2 file, then it's better to use qemu-nbd since it knows about the qcow2 format: $ qemu-nbd -t disk & $ qemu-img info nbd://localhost image: nbd://localhost:10809 file format: raw virtual size: 1.0G (1073741824 bytes) disk size: unavailable Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Wed, Nov 21, 2018 at 09:25:05AM +0000, Richard W.M. Jones wrote:
nbdkit: file.2: error: invalid request: offset and count are out of range: offset=196608 count=512
Actually what happens even more precisely is that the underlying file is not a multiple of 512 bytes (196624 ≡ 16 mod 512). qemu-img (the client) issues a request for the final "sector" of the file which goes beyond the end of the file. You can "fix" this by using the truncate filter (https://github.com/libguestfs/nbdkit/blob/master/filters/truncate/nbdkit-tru...) telling it to round up the size to the next multiple of 512 bytes: $ nbdkit --filter=truncate file file=disk -f -v round-up=512 and this allows qemu-img to work: $ qemu-img info nbd://localhost image: nbd://localhost:10809 file format: qcow2 virtual size: 1.0G (1073741824 bytes) disk size: unavailable cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false However the fundamental problem remains that you're exporting qcow2 bytes over the NBD connection which is not really what you wanted to do. You should use qemu-nbd instead since it understands qcow2 natively, or use raw format disks if you want to use nbdkit for its other features. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 11/21/18 4:39 AM, Richard W.M. Jones wrote:
On Wed, Nov 21, 2018 at 09:25:05AM +0000, Richard W.M. Jones wrote:
nbdkit: file.2: error: invalid request: offset and count are out of range: offset=196608 count=512
Actually what happens even more precisely is that the underlying file is not a multiple of 512 bytes (196624 ≡ 16 mod 512).
qemu-img (the client) issues a request for the final "sector" of the file which goes beyond the end of the file.
I'm still hoping to fix that bug in qemu before the 3.1 release, but that date is getting close :)
You can "fix" this by using the truncate filter (https://github.com/libguestfs/nbdkit/blob/master/filters/truncate/nbdkit-tru...) telling it to round up the size to the next multiple of 512 bytes:
$ nbdkit --filter=truncate file file=disk -f -v round-up=512
and this allows qemu-img to work:
Yes, this is the right workaround for now.
$ qemu-img info nbd://localhost image: nbd://localhost:10809 file format: qcow2
Invoking qemu-img without '-f $format' means that qemu-img has to probe the image to guess its format. Here, the probe succeeded, but in general this is insecure (if the file was 'raw', but the probe returned anything else, then a malicious guest could have tried to write metadata into the initial sector of their disk to try and get qemu to perform something on the guest's behalf due to the incorrect probe). A more robust solution pairs either: qemu-nbd -f raw file.qcow2 <=> qemu-img info -f qcow2 nbd://localhost nbdkit file file.qcow2 <=> qemu-img info -f qcow2 nbd://localhost where NBD is exposing the qcow2 metadata over the network (whether via qemu-nbd or nbdkit), and the client HAS to be aware of qcow2, or: qemu-nbd -f qcow2 file.qcow2 <=> qemu-img info -f raw nbd://localhost qemu-nbd -f qcow2 file.qcow2 <=> any other nbd client where qemu-nbd is interpreting the qcow2 file, and only exposing the guest-visible data over the network. Or, repeating the above in a different formulation - when you are interested in serving qcow2 data over the network, you HAVE to have exactly one of your two endpoints be aware that the qcow2 format is involved - either the server is agnostic and the client interprets qcow2 (the former setup), or the server interprets qcow2 and the client is agnostic (the latter setup). Then, as Rich pointed out, when you expose qcow2 over NBD (the former setup), the current NBD protocol does NOT allow resizes - so while reading the image works okay, writing to the image is likely to hit ENOSPC unless you pre-allocated the .qcow2 file being served. That is likely to change upstream in the next six months (it's on my todo list to get NBD resize implemented), but there's still a percolation delay from upstream to using it in default OS installs. On the other hand, when you expose raw over NBD by using qemu-nbd to convert qcow2 to raw on the server end, then the server can resize things as needed and the client need not be aware that qcow2 is even involved. As for why nbdkit doesn't have a plugin for exposing a local qcow2 file as raw - we don't see the point in duplicating what qemu-nbd can already do. Yes, someone could write such a plugin, but it will probably have worse performance than what you can already get out of using qemu-nbd as the server.
However the fundamental problem remains that you're exporting qcow2 bytes over the NBD connection which is not really what you wanted to do.
Maybe it is, if the qcow2 file is fully preallocated, but yeah, until NBD resize exists, most people don't want that combination.
You should use qemu-nbd instead since it understands qcow2 natively, or use raw format disks if you want to use nbdkit for its other features.
Rich.
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Wed, Nov 21, 2018 at 10:33 PM Eric Blake <eblake@redhat.com> wrote:
On 11/21/18 4:39 AM, Richard W.M. Jones wrote:
On Wed, Nov 21, 2018 at 09:25:05AM +0000, Richard W.M. Jones wrote:
nbdkit: file.2: error: invalid request: offset and count are out of range: offset=196608 count=512
Actually what happens even more precisely is that the underlying file is not a multiple of 512 bytes (196624 ≡ 16 mod 512).
qemu-img (the client) issues a request for the final "sector" of the file which goes beyond the end of the file.
I'm still hoping to fix that bug in qemu before the 3.1 release, but that date is getting close :)
You can "fix" this by using the truncate filter (
https://github.com/libguestfs/nbdkit/blob/master/filters/truncate/nbdkit-tru... )
telling it to round up the size to the next multiple of 512 bytes:
$ nbdkit --filter=truncate file file=disk -f -v round-up=512
and this allows qemu-img to work:
Yes, this is the right workaround for now.
$ qemu-img info nbd://localhost image: nbd://localhost:10809 file format: qcow2
Invoking qemu-img without '-f $format' means that qemu-img has to probe the image to guess its format. Here, the probe succeeded, but in general this is insecure (if the file was 'raw', but the probe returned anything else, then a malicious guest could have tried to write metadata into the initial sector of their disk to try and get qemu to perform something on the guest's behalf due to the incorrect probe). A more robust solution pairs either:
qemu-nbd -f raw file.qcow2 <=> qemu-img info -f qcow2 nbd://localhost nbdkit file file.qcow2 <=> qemu-img info -f qcow2 nbd://localhost
where NBD is exposing the qcow2 metadata over the network (whether via qemu-nbd or nbdkit), and the client HAS to be aware of qcow2, or:
qemu-nbd -f qcow2 file.qcow2 <=> qemu-img info -f raw nbd://localhost qemu-nbd -f qcow2 file.qcow2 <=> any other nbd client
where qemu-nbd is interpreting the qcow2 file, and only exposing the guest-visible data over the network.
Or, repeating the above in a different formulation - when you are interested in serving qcow2 data over the network, you HAVE to have exactly one of your two endpoints be aware that the qcow2 format is involved - either the server is agnostic and the client interprets qcow2 (the former setup), or the server interprets qcow2 and the client is agnostic (the latter setup).
Then, as Rich pointed out, when you expose qcow2 over NBD (the former setup), the current NBD protocol does NOT allow resizes - so while reading the image works okay, writing to the image is likely to hit ENOSPC unless you pre-allocated the .qcow2 file being served. That is likely to change upstream in the next six months (it's on my todo list to get NBD resize implemented), but there's still a percolation delay from upstream to using it in default OS installs. On the other hand, when you expose raw over NBD by using qemu-nbd to convert qcow2 to raw on the server end, then the server can resize things as needed and the client need not be aware that qcow2 is even involved.
After nbd resize implemented, we can get actual disk size instread of 'unavailable', right? Will it support disk extend, or just support disk shrinking?
As for why nbdkit doesn't have a plugin for exposing a local qcow2 file as raw - we don't see the point in duplicating what qemu-nbd can already do. Yes, someone could write such a plugin, but it will probably have worse performance than what you can already get out of using qemu-nbd as the server.
However the fundamental problem remains that you're exporting qcow2 bytes over the NBD connection which is not really what you wanted to do.
Maybe it is, if the qcow2 file is fully preallocated, but yeah, until NBD resize exists, most people don't want that combination.
You should use qemu-nbd instead since it understands qcow2 natively, or use raw format disks if you want to use nbdkit for its other features.
Rich.
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
-- Best regards, ----------------------------------- Han Han Quality Engineer Redhat. Email: hhan@redhat.com Phone: +861065339333
On 11/23/18 3:56 AM, Han Han wrote:
Then, as Rich pointed out, when you expose qcow2 over NBD (the former setup), the current NBD protocol does NOT allow resizes - so while reading the image works okay, writing to the image is likely to hit ENOSPC unless you pre-allocated the .qcow2 file being served. That is likely to change upstream in the next six months (it's on my todo list to get NBD resize implemented), but there's still a percolation delay from upstream to using it in default OS installs. On the other hand, when you expose raw over NBD by using qemu-nbd to convert qcow2 to raw on the server end, then the server can resize things as needed and the client need not be aware that qcow2 is even involved.
After nbd resize implemented, we can get actual disk size instread of 'unavailable', right?
qemu-img will probably always report 'disk size: unavailable' for NBD exports. For regular raw files, it obtains that information by calling stat(), but the NBD protocol does not have an equivalent to stat() for showing how much disk space an image occupies. But you can safely ignore that line being listed as 'unavailable', because by the time you are talking to a remote server, it is the remote server's problem about how much space the image occupies, and not yours, because the image doesn't occupy any space locally.
Will it support disk extend, or just support disk shrinking?
Whether the extension will support extension and/or shrinking will depend on the server. Most likely, there will be servers that support extension but not shrinking (as it is much easier to extend than it is to worry about whether a shrink will truncate important data). And, the point of being an extension means that a client should be prepared for a server that does not support resizes, or that some resize requests will fail even if others can succeed. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
participants (3)
-
Eric Blake -
Han Han -
Richard W.M. Jones