9:33 p.m.
Hello, I face a problem on the nbdkit server:
Version-Release number of selected component (if applicable):
nbdkit-v1.9.1
qemu-utils-2.12+dfsg-3+b1
How reproducible:
100%
Steps:
1. Create a qcow2 file
# qemu-img create disk 1G -f qcow2
2. Export the qcow2 file then try to read it via nbd url
# nbdkit file file=disk -e raw -v
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: debug: registering plugin
/usr/local/lib/nbdkit/plugins/nbdkit-file-plugin.so
nbdkit: debug: registered plugin
/usr/local/lib/nbdkit/plugins/nbdkit-file-plugin.so (name file)
nbdkit: debug: file: load
nbdkit: debug: file: config key=file, value=/tmp/disk
nbdkit: debug: file: config_complete
nbdkit: debug: bound to IP address <any>:10809 (2 socket(s))
nbdkit: debug: forked into background (new pid = 32524)
3. Try to read the file via nbd:
# qemu-img info
nbd://localhost/raw
qemu-img: Could not open 'nbd://localhost/raw': Could not read L1 table:
Invalid argument
Logs from nbdkit:
nbdkit: debug: file: open readonly=0
nbdkit: file[1]: debug: newstyle negotiation: flags: global 0x3
nbdkit: file[1]: debug: newstyle negotiation: client flags: 0x3
nbdkit: file[1]: debug: newstyle negotiation: NBD_OPT_GO: client requested
export 'raw' (ignored)
nbdkit: file[1]: debug: get_size
nbdkit: file[1]: debug: can_write
nbdkit: file[1]: debug: can_zero
nbdkit: file[1]: debug: can_write
nbdkit: file[1]: debug: can_trim
nbdkit: file[1]: debug: can_fua
nbdkit: file[1]: debug: can_flush
nbdkit: file[1]: debug: is_rotational
nbdkit: file[1]: debug: newstyle negotiation: flags: export 0x6d
nbdkit: file[1]: debug: newstyle negotiation: NBD_OPT_GO: ignoring
NBD_INFO_* request 3
nbdkit: file[1]: debug: handshake complete, processing requests with 16
threads
nbdkit: debug: starting worker thread file.0
nbdkit: debug: starting worker thread file.1
nbdkit: debug: starting worker thread file.3
nbdkit: debug: starting worker thread file.2
nbdkit: debug: starting worker thread file.4
nbdkit: debug: starting worker thread file.5
nbdkit: debug: starting worker thread file.7
nbdkit: debug: starting worker thread file.6
nbdkit: debug: starting worker thread file.8
nbdkit: debug: starting worker thread file.9
nbdkit: debug: starting worker thread file.10
nbdkit: debug: starting worker thread file.11
nbdkit: debug: starting worker thread file.12
nbdkit: debug: starting worker thread file.13
nbdkit: debug: starting worker thread file.14
nbdkit: debug: starting worker thread file.15
nbdkit: file.1: debug: pread count=512 offset=0
nbdkit: file.0: debug: pread count=512 offset=0
nbdkit: file.3: debug: sending error reply: Invalid argument
nbdkit: file.4: debug: client sent disconnect command, closing connection
nbdkit: file.4: debug: exiting worker thread file.4
nbdkit: file.2: debug: exiting worker thread file.2
nbdkit: file.7: debug: exiting worker thread file.7
nbdkit: file.5: debug: exiting worker thread file.5
nbdkit: file.6: debug: exiting worker thread file.6
nbdkit: file.8: debug: exiting worker thread file.8
nbdkit: file.9: debug: exiting worker thread file.9
nbdkit: file.10: debug: exiting worker thread file.10
nbdkit: file.11: debug: exiting worker thread file.11
nbdkit: file.12: debug: exiting worker thread file.12
nbdkit: file.0: debug: exiting worker thread file.0
nbdkit: file.14: debug: exiting worker thread file.14
nbdkit: file.15: debug: exiting worker thread file.15
nbdkit: file.1: debug: exiting worker thread file.1
nbdkit: file.13: debug: exiting worker thread file.13
nbdkit: file.3: debug: exiting worker thread file.3
nbdkit: file[1]: debug: close
Could you please check if it is a bug from nbdkit? Thanks
--
Best regards,
-----------------------------------
Han Han
Quality Engineer
Redhat.
Email: hhan(a)redhat.com
Phone: +861065339333
3:25 a.m.
On Wed, Nov 21, 2018 at 11:33:33AM +0800, Han Han wrote:
Hello, I face a problem on the nbdkit server:
Version-Release number of selected component (if applicable):
nbdkit-v1.9.1
qemu-utils-2.12+dfsg-3+b1
How reproducible:
100%
Steps:
1. Create a qcow2 file
# qemu-img create disk 1G -f qcow2
This creates a 1GB virtual size qcow2 (real size is about 196K):
$ ll disk
-rw-rw-r--. 1 rjones rjones 196624 Nov 21 09:20 disk
$ file disk
disk: QEMU QCOW Image (v3), 1073741824 bytes
2. Export the qcow2 file then try to read it via nbd url
# nbdkit file file=disk -e raw -v
This exports the disk using the export name "raw". However nbdkit
doesn't know anything about qcow2, so this exports bytes of qcow2 data
to the client. The virtual size of the disk will be about 196K, and
it won't be resizable (so you cannot write to it).
Note that the -e (export name) option is not used by nbdkit so you
don't need it.
3. Try to read the file via nbd:
# qemu-img info
nbd://localhost/raw
What precisely happens here is that qemu-img sees a qcow2 file and
tries to read beyond the end of the file:
nbdkit: file.2: error: invalid request: offset and count are out of range: offset=196608
count=512
If you want to export a qcow2 file, then it's better to use qemu-nbd
since it knows about the qcow2 format:
$ qemu-nbd -t disk &
$ qemu-img info nbd://localhost
image: nbd://localhost:10809
file format: raw
virtual size: 1.0G (1073741824 bytes)
disk size: unavailable
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
4:39 a.m.
On Wed, Nov 21, 2018 at 09:25:05AM +0000, Richard W.M. Jones wrote:
nbdkit: file.2: error: invalid request: offset and count are out of
range: offset=196608 count=512
Actually what happens even more precisely is that the underlying file
is not a multiple of 512 bytes (196624 ≡ 16 mod 512).
qemu-img (the client) issues a request for the final "sector" of the
file which goes beyond the end of the file.
You can "fix" this by using the truncate filter
(https://github.com/libguestfs/nbdkit/blob/master/filters/truncate/nbdkit-...)
telling it to round up the size to the next multiple of 512 bytes:
$ nbdkit --filter=truncate file file=disk -f -v round-up=512
and this allows qemu-img to work:
$ qemu-img info nbd://localhost
image: nbd://localhost:10809
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: unavailable
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
However the fundamental problem remains that you're exporting qcow2
bytes over the NBD connection which is not really what you wanted to
do.
You should use qemu-nbd instead since it understands qcow2 natively,
or use raw format disks if you want to use nbdkit for its other
features.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
8:33 a.m.
On 11/21/18 4:39 AM, Richard W.M. Jones wrote:
On Wed, Nov 21, 2018 at 09:25:05AM +0000, Richard W.M. Jones wrote:
> nbdkit: file.2: error: invalid request: offset and count are out of range:
offset=196608 count=512
Actually what happens even more precisely is that the underlying file
is not a multiple of 512 bytes (196624 ≡ 16 mod 512).
qemu-img (the client) issues a request for the final "sector" of the
file which goes beyond the end of the file.
I'm still hoping to fix that bug in qemu before the 3.1 release, but
that date is getting close :)
You can "fix" this by using the truncate filter
(https://github.com/libguestfs/nbdkit/blob/master/filters/truncate/nbdkit-...)
telling it to round up the size to the next multiple of 512 bytes:
$ nbdkit --filter=truncate file file=disk -f -v round-up=512
and this allows qemu-img to work:
Yes, this is the right workaround for now.
$ qemu-img info nbd://localhost
image: nbd://localhost:10809
file format: qcow2
Invoking qemu-img without '-f $format' means that qemu-img has to probe
the image to guess its format. Here, the probe succeeded, but in general
this is insecure (if the file was 'raw', but the probe returned anything
else, then a malicious guest could have tried to write metadata into the
initial sector of their disk to try and get qemu to perform something on
the guest's behalf due to the incorrect probe). A more robust solution
pairs either:
qemu-nbd -f raw file.qcow2 <=> qemu-img info -f qcow2 nbd://localhost
nbdkit file file.qcow2 <=> qemu-img info -f qcow2 nbd://localhost
where NBD is exposing the qcow2 metadata over the network (whether via
qemu-nbd or nbdkit), and the client HAS to be aware of qcow2, or:
qemu-nbd -f qcow2 file.qcow2 <=> qemu-img info -f raw nbd://localhost
qemu-nbd -f qcow2 file.qcow2 <=> any other nbd client
where qemu-nbd is interpreting the qcow2 file, and only exposing the
guest-visible data over the network.
Or, repeating the above in a different formulation - when you are
interested in serving qcow2 data over the network, you HAVE to have
exactly one of your two endpoints be aware that the qcow2 format is
involved - either the server is agnostic and the client interprets qcow2
(the former setup), or the server interprets qcow2 and the client is
agnostic (the latter setup).
Then, as Rich pointed out, when you expose qcow2 over NBD (the former
setup), the current NBD protocol does NOT allow resizes - so while
reading the image works okay, writing to the image is likely to hit
ENOSPC unless you pre-allocated the .qcow2 file being served. That is
likely to change upstream in the next six months (it's on my todo list
to get NBD resize implemented), but there's still a percolation delay
from upstream to using it in default OS installs. On the other hand,
when you expose raw over NBD by using qemu-nbd to convert qcow2 to raw
on the server end, then the server can resize things as needed and the
client need not be aware that qcow2 is even involved.
As for why nbdkit doesn't have a plugin for exposing a local qcow2 file
as raw - we don't see the point in duplicating what qemu-nbd can already
do. Yes, someone could write such a plugin, but it will probably have
worse performance than what you can already get out of using qemu-nbd as
the server.
However the fundamental problem remains that you're exporting
qcow2
bytes over the NBD connection which is not really what you wanted to
do.
Maybe it is, if the qcow2 file is fully preallocated, but yeah, until
NBD resize exists, most people don't want that combination.
You should use qemu-nbd instead since it understands qcow2 natively,
or use raw format disks if you want to use nbdkit for its other
features.
Rich.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
3:56 a.m.
On Wed, Nov 21, 2018 at 10:33 PM Eric Blake <eblake(a)redhat.com> wrote:
On 11/21/18 4:39 AM, Richard W.M. Jones wrote:
> On Wed, Nov 21, 2018 at 09:25:05AM +0000, Richard W.M. Jones wrote:
>> nbdkit: file.2: error: invalid request: offset and count are out of
range: offset=196608 count=512
>
> Actually what happens even more precisely is that the underlying file
> is not a multiple of 512 bytes (196624 ≡ 16 mod 512).
>
> qemu-img (the client) issues a request for the final "sector" of the
> file which goes beyond the end of the file.
I'm still hoping to fix that bug in qemu before the 3.1 release, but
that date is getting close :)
>
> You can "fix" this by using the truncate filter
> (
https://github.com/libguestfs/nbdkit/blob/master/filters/truncate/nbdkit-...
)
> telling it to round up the size to the next multiple of 512 bytes:
>
> $ nbdkit --filter=truncate file file=disk -f -v round-up=512
>
> and this allows qemu-img to work:
Yes, this is the right workaround for now.
>
> $ qemu-img info nbd://localhost
> image: nbd://localhost:10809
> file format: qcow2
Invoking qemu-img without '-f $format' means that qemu-img has to probe
the image to guess its format. Here, the probe succeeded, but in general
this is insecure (if the file was 'raw', but the probe returned anything
else, then a malicious guest could have tried to write metadata into the
initial sector of their disk to try and get qemu to perform something on
the guest's behalf due to the incorrect probe). A more robust solution
pairs either:
qemu-nbd -f raw file.qcow2 <=> qemu-img info -f qcow2 nbd://localhost
nbdkit file file.qcow2 <=> qemu-img info -f qcow2 nbd://localhost
where NBD is exposing the qcow2 metadata over the network (whether via
qemu-nbd or nbdkit), and the client HAS to be aware of qcow2, or:
qemu-nbd -f qcow2 file.qcow2 <=> qemu-img info -f raw nbd://localhost
qemu-nbd -f qcow2 file.qcow2 <=> any other nbd client
where qemu-nbd is interpreting the qcow2 file, and only exposing the
guest-visible data over the network.
Or, repeating the above in a different formulation - when you are
interested in serving qcow2 data over the network, you HAVE to have
exactly one of your two endpoints be aware that the qcow2 format is
involved - either the server is agnostic and the client interprets qcow2
(the former setup), or the server interprets qcow2 and the client is
agnostic (the latter setup).
Then, as Rich pointed out, when you expose qcow2 over NBD (the former
setup), the current NBD protocol does NOT allow resizes - so while
reading the image works okay, writing to the image is likely to hit
ENOSPC unless you pre-allocated the .qcow2 file being served. That is
likely to change upstream in the next six months (it's on my todo list
to get NBD resize implemented), but there's still a percolation delay
from upstream to using it in default OS installs. On the other hand,
when you expose raw over NBD by using qemu-nbd to convert qcow2 to raw
on the server end, then the server can resize things as needed and the
client need not be aware that qcow2 is even involved.
After nbd resize implemented, we can get actual disk size instread of
'unavailable', right?
Will it support disk extend, or just support disk shrinking?
As for why nbdkit doesn't have a plugin for exposing a local
qcow2 file
as raw - we don't see the point in duplicating what qemu-nbd can already
do. Yes, someone could write such a plugin, but it will probably have
worse performance than what you can already get out of using qemu-nbd as
the server.
> However the fundamental problem remains that you're exporting qcow2
> bytes over the NBD connection which is not really what you wanted to
> do.
Maybe it is, if the qcow2 file is fully preallocated, but yeah, until
NBD resize exists, most people don't want that combination.
>
> You should use qemu-nbd instead since it understands qcow2 natively,
> or use raw format disks if you want to use nbdkit for its other
> features.
>
> Rich.
>
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
--
Best regards,
-----------------------------------
Han Han
Quality Engineer
Redhat.
Email: hhan(a)redhat.com
Phone: +861065339333
12:06 p.m.
On 11/23/18 3:56 AM, Han Han wrote:
> Then, as Rich pointed out, when you expose qcow2 over NBD (the
former
> setup), the current NBD protocol does NOT allow resizes - so while
> reading the image works okay, writing to the image is likely to hit
> ENOSPC unless you pre-allocated the .qcow2 file being served. That is
> likely to change upstream in the next six months (it's on my todo list
> to get NBD resize implemented), but there's still a percolation delay
> from upstream to using it in default OS installs. On the other hand,
> when you expose raw over NBD by using qemu-nbd to convert qcow2 to raw
> on the server end, then the server can resize things as needed and the
> client need not be aware that qcow2 is even involved.
>
> After nbd resize implemented, we can get actual disk size instread of
'unavailable', right?
qemu-img will probably always report 'disk size: unavailable' for NBD
exports. For regular raw files, it obtains that information by calling
stat(), but the NBD protocol does not have an equivalent to stat() for
showing how much disk space an image occupies. But you can safely
ignore that line being listed as 'unavailable', because by the time you
are talking to a remote server, it is the remote server's problem about
how much space the image occupies, and not yours, because the image
doesn't occupy any space locally.
Will it support disk extend, or just support disk shrinking?
Whether the extension will support extension and/or shrinking will
depend on the server. Most likely, there will be servers that support
extension but not shrinking (as it is much easier to extend than it is
to worry about whether a shrink will truncate important data). And, the
point of being an extension means that a client should be prepared for a
server that does not support resizes, or that some resize requests will
fail even if others can succeed.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
2069
days inactive
2074
days old
5 comments
3 participants
participants (3)
-
Eric Blake -
Han Han -
Richard W.M. Jones