Plans for nbdkit 1.18 release?
Hi Eric, nbdkit 1.16 was released over 3 months ago, and there has been quite a lot of upstream development since then, so it could be about time for a new stable release. Anything in particular that you would like to get in to 1.18? Or anything we've added that you have reservations about supporting long-term? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On 2/20/20 3:37 PM, Richard W.M. Jones wrote:
Hi Eric,
nbdkit 1.16 was released over 3 months ago, and there has been quite a lot of upstream development since then, so it could be about time for a new stable release. Anything in particular that you would like to get in to 1.18? Or anything we've added that you have reservations about supporting long-term?
My biggest worry is whether we want to make python API version 2 plugins support an optional flags parameter. I think we decided that for version 1 plugins, making .zero's may_trim argument optional was not worth it. But especially given that things like .pread currently always pass a flags=0 parameter, making it optional for API version 2 is still worth pursuing. As you recall, I did post potential glue code for optional parameters, but it was different than the code we eventually took when adding API version 2 support, and I have not yet had time to rebase the salvageable parts of my patches. On the other hand, releasing 1.18 with mandatory flags= parameter in python plugins, and later relaxing it when we figure out the glue code, shouldn't be a problem. It is not backwards incompatible to make a once-required parameter become optional. So I don't see it as holding up a release. My proposal to add an NBD_INFO_INIT_STATE extension is still pending on multiple fronts (approval at the NBD spec, and a rewrite of both qemu and nbdkit patches to address review comments); I'm not worried if that misses 1.18. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Thu, Feb 20, 2020 at 05:03:18PM -0600, Eric Blake wrote:
On 2/20/20 3:37 PM, Richard W.M. Jones wrote:
Hi Eric,
nbdkit 1.16 was released over 3 months ago, and there has been quite a lot of upstream development since then, so it could be about time for a new stable release. Anything in particular that you would like to get in to 1.18? Or anything we've added that you have reservations about supporting long-term?
My biggest worry is whether we want to make python API version 2 plugins support an optional flags parameter. I think we decided that for version 1 plugins, making .zero's may_trim argument optional was not worth it. But especially given that things like .pread currently always pass a flags=0 parameter, making it optional for API version 2 is still worth pursuing. As you recall, I did post potential glue code for optional parameters, but it was different than the code we eventually took when adding API version 2 support, and I have not yet had time to rebase the salvageable parts of my patches.
I think making the flags parameter option, ie this: def pread(h, buf, offset, flags=0): def trim(h, count, offset, flags=0): could be worthwhile, although as you say below it we could still change this later. I don't think the bitmasks are bad, since plenty of standard Python APIs seem to take bitmasks, and they're also much faster to handle since they can be passed almost directly to the C API.
On the other hand, releasing 1.18 with mandatory flags= parameter in python plugins, and later relaxing it when we figure out the glue code, shouldn't be a problem. It is not backwards incompatible to make a once-required parameter become optional. So I don't see it as holding up a release.
My proposal to add an NBD_INFO_INIT_STATE extension is still pending on multiple fronts (approval at the NBD spec, and a rewrite of both qemu and nbdkit patches to address review comments); I'm not worried if that misses 1.18.
OK, thanks! I'll have a go at some release notes next week which should give us a clearer idea of what's new/changes. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Preliminary release notes: https://github.com/libguestfs/nbdkit/commit/10fa8e8d587575d0c107afec29bad7e2... -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Eric: Did you want to take this one any further? It might be one that we save for > 1.18: https://www.redhat.com/archives/libguestfs/2020-February/thread.html#00206 Another thing I've been thinking about for some time is splitting .config_complete into .config_complete + .get_ready (new name TBD). At the moment .config_complete is both the place where we finish processing config, and also the last chance we get to set things up before the server forks. Of course such a change would not be material — .get_ready would always be run immediately after .config_complete — but it cleans up the API a little. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On 2/22/20 4:37 AM, Richard W.M. Jones wrote:
Eric:
Did you want to take this one any further? It might be one that we save for > 1.18:
https://www.redhat.com/archives/libguestfs/2020-February/thread.html#00206
At this point, I'm thinking save it for post-1.18 (we've already got enough churn on VDDK with LD_LIBRARY_PATH changes).
Another thing I've been thinking about for some time is splitting .config_complete into .config_complete + .get_ready (new name TBD). At the moment .config_complete is both the place where we finish processing config, and also the last chance we get to set things up before the server forks. Of course such a change would not be material — .get_ready would always be run immediately after .config_complete — but it cleans up the API a little.
How would it play with --dump-config? Right now, we allow that to run after .config but before .config_complete, precisely so that language bindings can accept the first .config to determine their script name, and then consult the script for any further config help. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Sat, Feb 22, 2020 at 05:11:01AM -0600, Eric Blake wrote:
On 2/22/20 4:37 AM, Richard W.M. Jones wrote:
Another thing I've been thinking about for some time is splitting .config_complete into .config_complete + .get_ready (new name TBD). At the moment .config_complete is both the place where we finish processing config, and also the last chance we get to set things up before the server forks. Of course such a change would not be material — .get_ready would always be run immediately after .config_complete — but it cleans up the API a little.
How would it play with --dump-config? Right now, we allow that to run after .config but before .config_complete, precisely so that language bindings can accept the first .config to determine their script name, and then consult the script for any further config help.
I don't believe it affects it at all. Here are a couple of patches which show the general idea and pass all the tests. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 2/22/20 5:42 AM, Richard W.M. Jones wrote:
On Sat, Feb 22, 2020 at 05:11:01AM -0600, Eric Blake wrote:
On 2/22/20 4:37 AM, Richard W.M. Jones wrote:
Another thing I've been thinking about for some time is splitting .config_complete into .config_complete + .get_ready (new name TBD). At the moment .config_complete is both the place where we finish processing config, and also the last chance we get to set things up before the server forks. Of course such a change would not be material — .get_ready would always be run immediately after .config_complete — but it cleans up the API a little.
How would it play with --dump-config? Right now, we allow that to run after .config but before .config_complete, precisely so that language bindings can accept the first .config to determine their script name, and then consult the script for any further config help.
I don't believe it affects it at all. Here are a couple of patches which show the general idea and pass all the tests.
One thought I've had (especially with the ext2 filter fresh on my mind) - it would be nice to be able to create a filter that shares a common open of the plugin. For example, in ext2, we cannot support parallel calls to ext2fs_open() (since that is basically mounting the file system, but you can't have parallel mounters), as a result, .thread_model returns SERIALIZE_CONNECTIONS (a second client has to wait for the first to unmount). What I'd really like is for ext2's .config_complete to validate that all parameters are sane, then have .get_ready be passed both a next_open and next_ops pointer so it can call a single next_ops->open() to get a handle to the underlying plugin, mount the file system once; then ext2's .open would NOT call through its next_open() parameter (because it already has a shared open handle), and we could have .prepare do JUST the parallelizable work of ext2fs_namei() per client but reusing the already-shared plugin handle from .get_ready. That is, multiple clients can open different files in parallel based on exportname, or even open different handles to the same file, but ALL of the handles play nicely because they still occur within the context of a SINGLE mount point and plugin handle. In fact, if we do something like that, we may even be able to write a filter that ADDS parallelism on top of an existing serialized plugin (because to the plugin's point of view, there is only a single client, it is the filter that is multiplexing multiple actual clients into serialized plugin access). To some extent, I already experimented with multiplexed access in the 'nbdkit nbd shared=1' plugin, where that code opens a single libnbd connection during .config_complete reused by all the .open (but doing it in a plugin was different then doing it in a filter, since the plugin is not relying on .next_ops). And what I did in the nbd plugin is an example of code that nicely fits the split between real .config_complete argument validation and .get_ready libnbd connection. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
participants (2)
-
Eric Blake -
Richard W.M. Jones