virt-sysprep future (was: Re: [PATCH V5] virt-sysprep:add ipconfig for preparation)
What's the plan for virt-sysprep? It's an interesting proof-of-concept. People love it! But it has several shortcomings. As I said before, I think it should be spun off into a separate project, and possibly be rewritten (/bin/bash is a terrible language for writing complicated things). It really needs to support at least some Windows guests to some degree. Any thoughts on this? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
Hi Rich,
What's the plan for virt-sysprep?
It's an interesting proof-of-concept.
People love it!
But it has several shortcomings. As I said before, I think it should be spun off into a separate project, and possibly be rewritten (/bin/bash is a terrible language for writing complicated things). It really needs to support at least some Windows guests to some degree.
Any thoughts on this?
Yes, I also think it should be spun off into a separate project, and be rewritten. But I want you doing the splitting work if you have time, and then I can try to rewrite it and do further works. And I considered trying to use NBD though it. NBD has already used in qemu named qemu-nbd, support all the virtual disk image, and the most important thing is that it's much more faster than guestmount, because it doesn't need to start a guest. But NBD client must be used under root, because it uses /dev/nbdX, and need to install the kernel nbd module. What do you think Rich? Thanks, Wanlong Gao
Rich.
On Sat, Mar 17, 2012 at 10:02:11AM +0800, Wanlong Gao wrote:
Hi Rich,
What's the plan for virt-sysprep?
It's an interesting proof-of-concept.
People love it!
But it has several shortcomings. As I said before, I think it should be spun off into a separate project, and possibly be rewritten (/bin/bash is a terrible language for writing complicated things). It really needs to support at least some Windows guests to some degree.
Any thoughts on this?
Yes, I also think it should be spun off into a separate project, and be rewritten. But I want you doing the splitting work if you have time, and then I can try to rewrite it and do further works.
Ha ha, but you know I'm going to rewrite it in OCaml :-)
And I considered trying to use NBD though it. NBD has already used in qemu named qemu-nbd, support all the virtual disk image, and the most important thing is that it's much more faster than guestmount, because it doesn't need to start a guest. But NBD client must be used under root, because it uses /dev/nbdX, and need to install the kernel nbd module.
What do you think Rich?
With my RHEL hat on, NBD wouldn't be any use because we don't support it, so we couldn't ship such a modified virt-sysprep in RHEL. I'm not clear why NBD is useful though. nbd+kpartx is much more limited than libguestfs. If guestmount is slow for you, let's work out why it's slow and make it faster (it's not slow for _me_). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
On 03/17/2012 06:49 PM, Richard W.M. Jones wrote:
On Sat, Mar 17, 2012 at 10:02:11AM +0800, Wanlong Gao wrote:
Hi Rich,
What's the plan for virt-sysprep?
It's an interesting proof-of-concept.
People love it!
But it has several shortcomings. As I said before, I think it should be spun off into a separate project, and possibly be rewritten (/bin/bash is a terrible language for writing complicated things). It really needs to support at least some Windows guests to some degree.
Any thoughts on this?
Yes, I also think it should be spun off into a separate project, and be rewritten. But I want you doing the splitting work if you have time, and then I can try to rewrite it and do further works.
Ha ha, but you know I'm going to rewrite it in OCaml :-)
That's great, I'm waiting for reading your code. ;)
And I considered trying to use NBD though it. NBD has already used in qemu named qemu-nbd, support all the virtual disk image, and the most important thing is that it's much more faster than guestmount, because it doesn't need to start a guest. But NBD client must be used under root, because it uses /dev/nbdX, and need to install the kernel nbd module.
What do you think Rich?
With my RHEL hat on, NBD wouldn't be any use because we don't support it, so we couldn't ship such a modified virt-sysprep in RHEL.
Does RedHat has a plan to support NBD in RHEL in the future?
I'm not clear why NBD is useful though. nbd+kpartx is much more limited than libguestfs. If guestmount is slow for you, let's work out why it's slow and make it faster (it's not slow for _me_).
Hmm... but I still think that the concept of NBD is good. :-D Thanks, Wanlong Gao
Rich.
On Sat, Mar 17, 2012 at 06:59:34PM +0800, Wanlong Gao wrote:
Does RedHat has a plan to support NBD in RHEL in the future?
Not as far as I know. guestmount is fast enough for me. With inspection: $ time sh -c 'guestmount -a /dev/vg_pin/F16x64 -i /tmp/mnt; fusermount -u /tmp/mnt' real 0m9.999s user 0m0.081s sys 0m0.377s Without inspection: $ time sh -c 'guestmount -a /dev/vg_pin/F16x64 -m /dev/vg_f16x64/lv_root /tmp/mnt; fusermount -u /tmp/mnt' real 0m6.681s user 0m0.074s sys 0m0.339s Currently virt-sysprep runs libguestfs twice (once for inspection, once to mount). A virt-sysprep that was rewritten not to be a shell script would only run libguestfs once, so there would be ~10 second overhead. This overhead is insignificant compared to the downsides of using nbd: having to run as root, (in)security, not supporting all guests, creating host device nodes, etc. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On 03/17/2012 07:49 PM, Richard W.M. Jones wrote:
On Sat, Mar 17, 2012 at 06:59:34PM +0800, Wanlong Gao wrote:
Does RedHat has a plan to support NBD in RHEL in the future?
Not as far as I know.
guestmount is fast enough for me. With inspection:
$ time sh -c 'guestmount -a /dev/vg_pin/F16x64 -i /tmp/mnt; fusermount -u /tmp/mnt'
real 0m9.999s user 0m0.081s sys 0m0.377s
Without inspection:
$ time sh -c 'guestmount -a /dev/vg_pin/F16x64 -m /dev/vg_f16x64/lv_root /tmp/mnt; fusermount -u /tmp/mnt'
real 0m6.681s user 0m0.074s sys 0m0.339s
Currently virt-sysprep runs libguestfs twice (once for inspection, once to mount). A virt-sysprep that was rewritten not to be a shell script would only run libguestfs once, so there would be ~10 second overhead. This overhead is insignificant compared to the downsides of using nbd: having to run as root, (in)security, not supporting all guests, creating host device nodes, etc.
[root@Allen ~]# time sh -c 'qemu-nbd -c /dev/nbd0 /dev/sda6; mount /dev/nbd0 ./tmp; umount ./tmp' real 0m0.146s user 0m0.009s sys 0m0.005s yes, "having run as root" is one of the shortcomings. I can't get why nbd causes security problem, do you mean data corruptions in muti-access? I can't understand the "not supporting all guest", since libguestfs points to look into the guest disk image, the concept of just read the disk images' format head is better. Thanks, Wanlong Gao
Rich.
On Sat, Mar 17, 2012 at 08:16:03PM +0800, Wanlong Gao wrote:
On 03/17/2012 07:49 PM, Richard W.M. Jones wrote:
On Sat, Mar 17, 2012 at 06:59:34PM +0800, Wanlong Gao wrote:
Does RedHat has a plan to support NBD in RHEL in the future?
Not as far as I know.
guestmount is fast enough for me. With inspection:
$ time sh -c 'guestmount -a /dev/vg_pin/F16x64 -i /tmp/mnt; fusermount -u /tmp/mnt'
real 0m9.999s user 0m0.081s sys 0m0.377s
Without inspection:
$ time sh -c 'guestmount -a /dev/vg_pin/F16x64 -m /dev/vg_f16x64/lv_root /tmp/mnt; fusermount -u /tmp/mnt'
real 0m6.681s user 0m0.074s sys 0m0.339s
Currently virt-sysprep runs libguestfs twice (once for inspection, once to mount). A virt-sysprep that was rewritten not to be a shell script would only run libguestfs once, so there would be ~10 second overhead. This overhead is insignificant compared to the downsides of using nbd: having to run as root, (in)security, not supporting all guests, creating host device nodes, etc.
[root@Allen ~]# time sh -c 'qemu-nbd -c /dev/nbd0 /dev/sda6; mount /dev/nbd0 ./tmp; umount ./tmp'
real 0m0.146s user 0m0.009s sys 0m0.005s
You ought to add kpartx and vgscan in there, for a realistic guest. No, it still won't (quite) be 6-10 seconds. That time is the difference the two methods. The real time for a full virt-sysprep run will be many seconds or minutes.
yes, "having run as root" is one of the shortcomings.
I can't get why nbd causes security problem, do you mean data corruptions in muti-access?
NBD isn't a security issue, but mounting untrusted guest filesystems on your host kernel might be. There's nothing to protect your host (and other guests) from an exploit, and you need to trust that every filesystem driver in your host kernel is hardened against malicious filesystems. When you use the 'mount' command above with the '-t' option, blkid "helpfully" scanned the untrusted disk for a signature and mounted the guest using that filesystem type; it could have been *any* filesystem type, even really obscure and unexpected ones. Filesystems are only the beginning; LVM metadata is another rich source of problems/exploits. The argument against mounting on the host will be even stronger when we add the option of using libvirt with libguestfs. Then you'll be able to hide the untrusted guest in an SVirt [SELinux] container, with cgroups for resource control.
I can't understand the "not supporting all guest", since libguestfs points to look into the guest disk image, the concept of just read the disk images' format head is better.
Sure, but now try a guest that contains volume groups which conflict with the host. You can (nowadays[1]) mount such VGs using UUIDs, but you have to find the UUIDs first -- complicated -- and you have to trust that the guest's LVM metadata didn't name any of your host PVs, either accidentally or maliciously. Rich. [1] https://bugzilla.redhat.com/show_bug.cgi?id=207470 -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
On Sat, Mar 17, 2012 at 01:57:10PM +0000, Richard W.M. Jones wrote:
filesystems. When you use the 'mount' command above with the '-t'
s/with/without/ Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On 03/17/2012 09:57 PM, Richard W.M. Jones wrote:
On Sat, Mar 17, 2012 at 08:16:03PM +0800, Wanlong Gao wrote:
On 03/17/2012 07:49 PM, Richard W.M. Jones wrote:
On Sat, Mar 17, 2012 at 06:59:34PM +0800, Wanlong Gao wrote:
Does RedHat has a plan to support NBD in RHEL in the future?
Not as far as I know.
guestmount is fast enough for me. With inspection:
$ time sh -c 'guestmount -a /dev/vg_pin/F16x64 -i /tmp/mnt; fusermount -u /tmp/mnt'
real 0m9.999s user 0m0.081s sys 0m0.377s
Without inspection:
$ time sh -c 'guestmount -a /dev/vg_pin/F16x64 -m /dev/vg_f16x64/lv_root /tmp/mnt; fusermount -u /tmp/mnt'
real 0m6.681s user 0m0.074s sys 0m0.339s
Currently virt-sysprep runs libguestfs twice (once for inspection, once to mount). A virt-sysprep that was rewritten not to be a shell script would only run libguestfs once, so there would be ~10 second overhead. This overhead is insignificant compared to the downsides of using nbd: having to run as root, (in)security, not supporting all guests, creating host device nodes, etc.
[root@Allen ~]# time sh -c 'qemu-nbd -c /dev/nbd0 /dev/sda6; mount /dev/nbd0 ./tmp; umount ./tmp'
real 0m0.146s user 0m0.009s sys 0m0.005s
You ought to add kpartx and vgscan in there, for a realistic guest. No, it still won't (quite) be 6-10 seconds. That time is the difference the two methods. The real time for a full virt-sysprep run will be many seconds or minutes.
yes, "having run as root" is one of the shortcomings.
I can't get why nbd causes security problem, do you mean data corruptions in muti-access?
NBD isn't a security issue, but mounting untrusted guest filesystems on your host kernel might be. There's nothing to protect your host (and other guests) from an exploit, and you need to trust that every filesystem driver in your host kernel is hardened against malicious filesystems. When you use the 'mount' command above with the '-t' option, blkid "helpfully" scanned the untrusted disk for a signature and mounted the guest using that filesystem type; it could have been *any* filesystem type, even really obscure and unexpected ones. Filesystems are only the beginning; LVM metadata is another rich source of problems/exploits.
The argument against mounting on the host will be even stronger when we add the option of using libvirt with libguestfs. Then you'll be able to hide the untrusted guest in an SVirt [SELinux] container, with cgroups for resource control.
I can't understand the "not supporting all guest", since libguestfs points to look into the guest disk image, the concept of just read the disk images' format head is better.
Sure, but now try a guest that contains volume groups which conflict with the host. You can (nowadays[1]) mount such VGs using UUIDs, but you have to find the UUIDs first -- complicated -- and you have to trust that the guest's LVM metadata didn't name any of your host PVs, either accidentally or maliciously.
Thank you very much for explain so clearly Rich. I have got so much information from you about this. Now, I'm waiting for your new virt-sysprep. :-D Regards, Wanlong Gao
Rich.
On 03/17/2012 06:49 PM, Richard W.M. Jones wrote:
On Sat, Mar 17, 2012 at 10:02:11AM +0800, Wanlong Gao wrote:
Hi Rich,
What's the plan for virt-sysprep?
It's an interesting proof-of-concept.
People love it!
But it has several shortcomings. As I said before, I think it should be spun off into a separate project, and possibly be rewritten (/bin/bash is a terrible language for writing complicated things). It really needs to support at least some Windows guests to some degree.
Any thoughts on this?
Yes, I also think it should be spun off into a separate project, and be rewritten. But I want you doing the splitting work if you have time, and then I can try to rewrite it and do further works.
Ha ha, but you know I'm going to rewrite it in OCaml :-)
Rich, I'd like to know when will you work out the virt-sysprep in Ocaml ? :-D Thanks, Wanlong Gao
And I considered trying to use NBD though it. NBD has already used in qemu named qemu-nbd, support all the virtual disk image, and the most important thing is that it's much more faster than guestmount, because it doesn't need to start a guest. But NBD client must be used under root, because it uses /dev/nbdX, and need to install the kernel nbd module.
What do you think Rich?
With my RHEL hat on, NBD wouldn't be any use because we don't support it, so we couldn't ship such a modified virt-sysprep in RHEL.
I'm not clear why NBD is useful though. nbd+kpartx is much more limited than libguestfs. If guestmount is slow for you, let's work out why it's slow and make it faster (it's not slow for _me_).
Rich.
On Mon, Mar 26, 2012 at 03:59:04PM +0800, Wanlong Gao wrote:
I'd like to know when will you work out the virt-sysprep in Ocaml ? :-D
I meant to write something about our investigations last week. Here we go ... Requirements: * Compatible with the current virt-sysprep, but easier to maintain and faster. * It should work in the "local" and over-libguestfs cases, which means using FUSE. * The current version is a shell script, so it's customizable by simple editing. I'd like to keep some customizability in the new version. These requirements could all be satisfied if we had: * Some kinds of "mount-local" ability, ie. integrating FUSE with the API in some way. [i] * Custom scripts: virt-sysprep --script foo.sh # runs foo.sh on the guest The problem is how to do the "mount-local". We analyzed this last week on IRC and came up with two possible methods: (1) We add FUSE support directly to the API, linking with libfuse. This is essentially the same as what was outlined in link [i] back in December 2011. (2) We add an attach-method "fd:<N>" which allows the file descriptor of the appliance to be passed over to another process. In this case we would allow guestmount to be passed a file descriptor (eg. "guestmount --attach-fd=<NN>") which it would then use exclusively. The problem with (1) is that we are not confident that it's a good idea to use libfuse as a true library. It doesn't appear that anyone else is using it this way. The problem with (2) is that it's complex and isn't future compatible with remote libvirt support [ii]. I suspect that we'll need to implement both, and be prepared to throw away a few implementations, before we know really which is the best way to go about this. Rich. [i] https://www.redhat.com/archives/libguestfs/2011-December/thread.html#00136 [ii] https://www.redhat.com/archives/libvir-list/2011-September/msg01049.html -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On 03/26/2012 05:00 PM, Richard W.M. Jones wrote:
On Mon, Mar 26, 2012 at 03:59:04PM +0800, Wanlong Gao wrote:
I'd like to know when will you work out the virt-sysprep in Ocaml ? :-D
I meant to write something about our investigations last week. Here we go ...
Requirements:
* Compatible with the current virt-sysprep, but easier to maintain and faster.
* It should work in the "local" and over-libguestfs cases, which means using FUSE.
* The current version is a shell script, so it's customizable by simple editing. I'd like to keep some customizability in the new version.
yeah, please initial your design so I can help doing something. ;)
These requirements could all be satisfied if we had:
* Some kinds of "mount-local" ability, ie. integrating FUSE with the API in some way. [i]
* Custom scripts: virt-sysprep --script foo.sh # runs foo.sh on the guest
This should be the necessary feature.
The problem is how to do the "mount-local". We analyzed this last week on IRC and came up with two possible methods:
(1) We add FUSE support directly to the API, linking with libfuse. This is essentially the same as what was outlined in link [i] back in December 2011.
(2) We add an attach-method "fd:<N>" which allows the file descriptor of the appliance to be passed over to another process. In this case we would allow guestmount to be passed a file descriptor (eg. "guestmount --attach-fd=<NN>") which it would then use exclusively.
But I still can't understand (2), what's the "fd:<N>" used to do? Thanks, Wanlong Gao
The problem with (1) is that we are not confident that it's a good idea to use libfuse as a true library. It doesn't appear that anyone else is using it this way.
The problem with (2) is that it's complex and isn't future compatible with remote libvirt support [ii].
I suspect that we'll need to implement both, and be prepared to throw away a few implementations, before we know really which is the best way to go about this.
Rich.
[i] https://www.redhat.com/archives/libguestfs/2011-December/thread.html#00136
[ii] https://www.redhat.com/archives/libvir-list/2011-September/msg01049.html
On Mon, Mar 26, 2012 at 07:24:54PM +0800, Wanlong Gao wrote:
But I still can't understand (2), what's the "fd:<N>" used to do?
Have a look at: https://github.com/libguestfs/libguestfs/blob/master/TODO#L624 However I don't think this approach is really feasible, because it would break any future effort to proxy the appliance connection over libvirt. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On Mon, Mar 26, 2012 at 10:00:53AM +0100, Richard W.M. Jones wrote:
(1) We add FUSE support directly to the API, linking with libfuse. This is essentially the same as what was outlined in link [i] back in December 2011. [...] The problem with (1) is that we are not confident that it's a good idea to use libfuse as a true library. It doesn't appear that anyone else is using it this way.
I'll also add that libfuse makes it tricky to use multiple handles. Specifically, FUSE callbacks get no context at all, eg: getattr's parameters are: static int getattr (const char *path, struct stat *statbuf); If libfuse could be called from multiple threads, each with its own handle, then how would you know which handle to use? (The answer possibly involves using thread-local storage, but that's hairy if either handle is multithreaded). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
On Mon, Mar 26, 2012 at 02:06:52PM +0100, Richard W.M. Jones wrote:
On Mon, Mar 26, 2012 at 10:00:53AM +0100, Richard W.M. Jones wrote:
(1) We add FUSE support directly to the API, linking with libfuse. This is essentially the same as what was outlined in link [i] back in December 2011. [...] The problem with (1) is that we are not confident that it's a good idea to use libfuse as a true library. It doesn't appear that anyone else is using it this way.
I'll also add that libfuse makes it tricky to use multiple handles.
Specifically, FUSE callbacks get no context at all, eg: getattr's parameters are:
static int getattr (const char *path, struct stat *statbuf);
If libfuse could be called from multiple threads, each with its own handle, then how would you know which handle to use? (The answer possibly involves using thread-local storage, but that's hairy if either handle is multithreaded).
Actually I'm wrong about this. Turns out there is a 'fuse_context' call that you can make within handlers which returns the context. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
participants (2)
-
Richard W.M. Jones -
Wanlong Gao