what size should NBD_OPT_INFO report?
by Eric Blake
I noticed during integration testing that nbd-server blindly reports a
size of 0 for all NBD_OPT_INFO requests, unless I pass a size argument
on the command line to nbd-server. At first, I thought it was a side
effect of me trying to use nbd-server on a block device (an LVM
partition), as it is a common bug to rely on stat().st_size which only
works for regular files (a block device has to use lseek(SEEK_END));
but then I noticed it happening when using nbd-server to serve regular
files as well.
I then turned to the source code, where I see that client->exportsize
is set in just these places:
commit_client()
- exportsize = OFFT_MAX, then try setupexport()
setupexport()
- default to client->server->expected_size (if one was provided),
further validating that actual size is large enough when actual size
can be computed
- if neither treefile or F_WAIT is set, compute actual size by opening
one or more files and using size_autodetect() (which does the right
thing for block devices, so my earlier thought about over-reliance
on stat() was wrong)
but these functions are only reached for NBD_OPT_EXPORT_NAME and
NBD_OPT_GO, not NBD_OPT_INFO. The upshot is that for NBD_OPT_GO,
there are some scenarios (treefile, F_WAIT) where nbd-server
advertises a size of 9223372036854775807 (0x7fffffff_ffffffff) meaning
unknown, but a size of 0 there is only possible if the file was
successfully opened and really is zero bytes in length. Conversely,
NBD_OPT_INFO is always advertising a size of 0, which means most of
the time, the size changes between NBD_OPT_INFO and NBD_OPT_GO.
For comparison, both nbdkit and qemu as an NBD server always advertise
the same size for both NBD_OPT_INFO and NBD_OPT_GO (but it should also
be noted that neither of these has the complexity of multifile like
nbd-server).
Should we tweak the NBD standard to recommend that a server advertise
a size of OFFT_MAX for NBD_OPT_INFO if it is prohibitive to determine
an actual size, rather than 0? Furthermore, is it worth adding code
to make NBD_OPT_INFO try harder to provide a sensible value when
possible (if expected_size was set, or if we are not multifile or
F_WAIT, then a stat() is enough to get the size for serving a regular
file; and if stat() says we have a block device, we can still try the
open/lseek/close)?
Also, is it worth trying to specify that since no known NBD servers
allow exports with sizes larger than OFFT_MAX, a server MUST NOT
report a size equal or larger than 9223372036854775808
(0x80000000_00000000)? There are definitely a few places that I could
simplify in libnbd if we have a protocol guarantee that a valid export
size will never have the most significant bit set, and therefore we
don't have to worry about whether size is represented as a signed or
unsigned value (which also implies that libnbd's function
nbd_get_size() returning ssize_t instead of size_t is acceptable).
But if we do that, then maybe 0x80000000_00000000 or even
0xffffffff_ffffffff would serve as a better recommended sentinel than
0x7fffffff_ffffffff for representing an indeterminate size.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization: qemu.org | libguestfs.org
4 days, 4 hours
Re: Setting up guestfs mirrors - v2
by Richard W.M. Jones
On Thu, Mar 28, 2024 at 10:46:45AM -0700, Luis Chamberlain wrote:
> Hey Richard,
>
> Sorry for the trouble but it seems my two emails got bounced.
> The first one I think becuase I went trigger happy and emailed
> before the list accepted me, and second I think because I was
> already on a moderated list from a prior email :P
>
> My goal was to email guestfs(a)lists.libguestfs.org and Cc
> kdevops(a)lists.linux.dev
Sorry, it should be working now! (Busy time and I'm on holiday ..)
> Luis
>
> On Thu, Mar 28, 2024 at 09:33:50AM -0700, Luis Chamberlain wrote:
> > [ v2 as I don't think the v1 email got through to guestfs list ]
> >
> > Thanks to the work by Jeff Layton, we've embraced guestfs on kdevops
> > [0] as a replacement to vagrant. To save bandwidth kdevops lets you
> > set up mirrors for all git trees we need, it would be nice to make it
> > easier for folks to also setup mirrors for guestfs base images, the
> > base images are here:
> >
> > https://builder.libguestfs.org/
> >
> > I take it we'd just add say
> > $XDG_CONFIG_HOME/virt-builder/repos.d/kdevops-local-mirror.conf with
> > the new URL where we established the mirror, is that correct?
Yes.
> > I was looking to see if there was an rsync mirror for the main URL
> > where all the images are above, or is there a better way to get the
> > latest updates?
Ideally we'd like to not host the images at all, but getting distros
to host them instead has not been very successful (except for SUSE).
> > Also, to save even more time with images, it would be useful if we had
> > base images getting updates a bit more regularly so that the step to
> > apt-get upgrade after bringup won't take too long by using the latest
> > base images. It seems this is possible with customer images, but it
> > would be nicer if we had this as a general thing. Any ideas to support
> > that?
Yes, we'd like Debian to generate their own images! Actually
they don't even need to generate images at all, just host
the index file.
Rich.
> > [0] https://github.com/linux-kdevops/kdevops
> >
> > Luis
> >
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
1 month, 2 weeks
Setting up guestfs mirrors - v2
by Luis Chamberlain
[ v2 as I don't think the v1 email got through to guestfs list ]
Thanks to the work by Jeff Layton, we've embraced guestfs on kdevops
[0] as a replacement to vagrant. To save bandwidth kdevops lets you
set up mirrors for all git trees we need, it would be nice to make it
easier for folks to also setup mirrors for guestfs base images, the
base images are here:
https://builder.libguestfs.org/
I take it we'd just add say
$XDG_CONFIG_HOME/virt-builder/repos.d/kdevops-local-mirror.conf with
the new URL where we established the mirror, is that correct?
I was looking to see if there was an rsync mirror for the main URL
where all the images are above, or is there a better way to get the
latest updates?
Also, to save even more time with images, it would be useful if we had
base images getting updates a bit more regularly so that the step to
apt-get upgrade after bringup won't take too long by using the latest
base images. It seems this is possible with customer images, but it
would be nicer if we had this as a general thing. Any ideas to support
that?
[0] https://github.com/linux-kdevops/kdevops
Luis
1 month, 3 weeks
Setting up guestfs mirrors
by Luis Chamberlain
Thanks to the work by Jeff Layton, we've embraced guestfs on kdevops
[0] as a replacement to vagrant. To save bandwidth kdevops lets you
set up mirrors for all git trees we need, it would be nice to make it
easier for folks to also setup mirrors for guestfs base images, the
base images are here:
https://builder.libguestfs.org/
I take it we'd just add say
$XDG_CONFIG_HOME/virt-builder/repos.d/kdevops-local-mirror.conf with
the new URL where we established the mirror, is that correct?
I was looking to see if there was an rsync mirror for the main URL
where all the images are above, or is there a better way to get the
latest updates?
Also, to save even more time with images, it would be useful if we had
base images getting updates a bit more regularly so that the step to
apt-get upgrade after bringup won't take too long by using the latest
base images. It seems this is possible with customer images, but it
would be nicer if we had this as a general thing. Any ideas to support
that?
[0] https://github.com/linux-kdevops/kdevops
Luis
1 month, 3 weeks
Re: problem with the scp protocol when using nbdkit
by Richard W.M. Jones
On Wed, Mar 27, 2024 at 08:20:10PM +0300, Виталя Иванво wrote:
[...]
> I'm faced with the problem of creating a block device with the SCP protocol.
> For example, I write 2 commands:
> 1. In the first terminal
> nbdkit --unix /tmp/usb1.sock --verbose --readonly curl url=scp://10.199.30.16/
> home/user/4Mfile user=user password=+/home/root/pass -D curl.verbose= 1 -f
If what you really want to do is access SSH servers, how about using
nbdkit-ssh-plugin instead? It is very well tested.
https://libguestfs.org/nbdkit-ssh-plugin.1.html
> 2. In the second terminal
> nbd-client -t 30 -u /tmp/usb1.sock /dev/nbd1 -n
>
> Output log:
> nbdkit: curl[1]: debug: newstyle negotiation: flags: export 0x103
> nbdkit: curl[1]: debug: replying to NBD_OPT_GO with NBD_REP_ACK
> nbdkit: curl[1]: debug: handshake complete, processing requests serially
> nbdkit: curl[1]: debug: curl: pread count=512 offset=0
> nbdkit: curl[1]: debug: Found bundle for host 10.199.30.16: 0x75d0bbd0
> [serially]
> nbdkit: curl[1]: debug: Re-using existing connection! (#0) with host
> 10.199.30.16
> nbdkit: curl[1]: debug: Connected to 10.199.30.16 (10.199.30.16) port 22 (#0)
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: <data with size=16384>
>
> ...
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: Connection #0 to host 10.199.30.16 left intact
> nbdkit: curl[1]: debug: curl: pread count=512 offset=2560
> nbdkit: curl[1]: debug: Found bundle for host 10.199.30.16: 0x75d0bbd0
> [serially]
> nbdkit: curl[1]: debug: Re-using existing connection! (#0) with host
> 10.199.30.16
> nbdkit: curl[1]: debug: Connected to 10.199.30.16 (10.199.30.16) port 22 (#0)
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: <data with size=16384>
>
> ...
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: <data with size=16384>
> nbdkit: curl[1]: debug: Connection #0 to host 10.199.30.16 left intact
> nbdkit: curl[1]: error: write reply: NBD_CMD_READ: Broken pipe
> nbdkit: curl[1]: debug: curl: finalize
> nbdkit: curl[1]: debug: curl: close
>
> The problem is that I cannot send a file larger than 2-3 megabytes. Not to
> mention that we want to send large images.
> I'm using the latest version of nbdkit 1.33.11.
> Such a breakdown only happens with STP, that is, FTP and other protocols can
> accept large files.
It must be something in curl. If you really want to use curl (not
nbdkit-ssh-plugin which I'd recommend instead), then you'll need to
enable curl debugging, ie. adding some of these flags:
https://libguestfs.org/nbdkit-curl-plugin.1.html#DEBUG-FLAGS
> Please tell me if there are patches that solve this problem.
> Thanks, Nikita Pavlov.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
1 month, 3 weeks
libnbd & nbdkit stable releases coming
by Richard W.M. Jones
libnbd & nbdkit stable releases were last done at the end of
September, so nearly 6 months ago. We have a healthy list of commits
in both projects since then (~ 100 in libnbd and ~ 135 in nbdkit), so
I plan to do a stable release of both towards the end of this month.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
2 months
[libguestfs BUG] ISO images can't be inspected properly
by Andrey Drobyshev
I discovered that when a VM has an ISO attached via CD-ROM, libguestfs
fails to read its partition table and therefore fails to perform basic
operations on a VM, e.g. listing a directory or editing a file:
# virsh domblklist rocky9-vm
Target Source
---------------------------------------------------------------------------
sda /vz/vmprivate/e827caad-b9e4-4858-b3dc-53ae6b8a5145/harddisk.hdd
sdb /vzt/images/Rocky-9.3-x86_64-minimal.iso
# virt-ls -d rocky9-vm /
libguestfs: error: inspect_os: sgdisk: Invalid partition data!
# virt-edit -d rocky9-vm /etc/fstab
libguestfs: error: inspect_os: sgdisk: Invalid partition data!
Even virt-inspector can't give us proper output on an ISO9660 image:
# virt-inspector /vzt/images/Rocky-9.3-x86_64-minimal.iso
libguestfs: error: inspect_os: sgdisk: Invalid partition data!
virt-inspector: no operating system could be detected inside this disk
image.
...
First I found a 3 years old topic where you suggested that sgdisk itself
might be too old:
https://listman.redhat.com/archives/libguestfs/2021-March/025939.html.
However this doesn't seem to be it in this case:
I've compiled the latest version from
https://sourceforge.net/projects/gptfdisk -- no luck as well. Seems
that sgdisk isn't capable of recognizing ISO9660 images.
Then I noticed 2 things:
1. libguestfs appliance is being created (with backend=libvirt) with ISO
attached as device='disk' (not 'cdrom'):
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2' cache='unsafe'/>
> <source file='/tmp/libguestfsD7IHvF/overlay2.qcow2' index='2'/>
> <backingStore type='file' index='5'>
> <format type='raw'/>
> <source file='/vzt/images/Rocky-9.3-x86_64-minimal.iso'/>
> <backingStore/>
> </backingStore>
> <target dev='sdb' bus='scsi'/>
> <alias name='scsi0-0-1-0'/>
> <address type='drive' controller='0' bus='0' target='1' unit='0'/>
> </disk>
That forces this disk to be supplied with "scsi-hd" driver in use, which
in turn leads to the device being visible inside the guest as a regular
SCSI hdd /dev/sdb. Shouldn't we, for instance, make sure that for such
images device='cdrom' and "scsi-cd" driver are being used? In this case
the guest will see it as /dev/sr0, and I suspect srX devices are being
skipped by guestfs_inspect_os.
2. When trying to perform the same operation via guestfish, everything
appears to be working:
# guestfish -a /vzt/images/Rocky-9.3-x86_64-minimal.iso -m /dev/sda ls /
>/dev/null && echo $?
0
File edit works as well. From debug output I see that sgdisk isn't
being invoked at all. That means there's a code path for the same
operations where sgdisk can be omitted. Maybe we should follow that
same code path when invoking other tools like virt-ls or virt-edit?
Overall, what would be an appropriate fix in your opinion? We should at
least make sure that working with domains (with the "-d" option) works
no matter what.
Andrey
2 months, 1 week
