6:03 a.m.
Hi,
recently we (oVirt) have started discussing whether the way virt-v2v
handles import from OVA files is good. And I would be interested in
ideas how it can be improved. It is likely somebody already gave some
thought to this problem.
TL;DR: Extracting the OVA before import is a problem for large VMs (in
sizes of TBs). Can we change something to prevent the extraction and
work directly over OVA?
What we consider a huge shortcoming is the fact that whole OVA is
extracted prior to the import into a temporary directory and processed
afterwards. Under normal situation user can have up to three copies of
the VM on his drive at the end of import:
* original OVA,
* temporary extracted files (will be deleted when virt-v2v terminates,
* converted VM.
This is not a good idea for large VMs that have hunderds of GBs or even
TBs in size. The requirements on the necessary storage space can be
lessened with proper partitioning. I.e. source OVA and converted VM
don't end up on the same drive and TMPDIR is set to put even temporary
files somewhere else. But this is not a general solution. And sometimes
the necessary space may not be available at all.
The question is how to change the import path so that virt-v2v doesn't
have to extract the OVA. I can see the following solutions:
1) Solve it virt-v2v: create a layer for directly accessing the files
in the archive.
2) Solve it in QEMU: create backing method that would allow creating
qemu disk backed by the archive.
3) Solve it on oVirt side: use some FUSE-based tool to provide
access to the archive and pass the OVA to virt-v2v not as a file but
as directory.
Does anyone have any other ideas or suggestions?
Best regards,
Tomas Golembiovsky
--
Tomáš Golembiovský <tgolembi(a)redhat.com>
7:02 a.m.
On Fri, Sep 09, 2016 at 01:03:49PM +0200, Tomáš Golembiovský wrote:
Hi,
recently we (oVirt) have started discussing whether the way virt-v2v
handles import from OVA files is good. And I would be interested in
ideas how it can be improved. It is likely somebody already gave some
thought to this problem.
TL;DR: Extracting the OVA before import is a problem for large VMs (in
sizes of TBs). Can we change something to prevent the extraction and
work directly over OVA?
Specifically virt-v2v needs to do:
qemu-img create -b <source-file-within-the-tarball> -f qcow2 overlay.qcow2
qemu-img convert overlay.qcow2 output
What we consider a huge shortcoming is the fact that whole OVA is
extracted prior to the import into a temporary directory and processed
afterwards. Under normal situation user can have up to three copies of
the VM on his drive at the end of import:
* original OVA,
* temporary extracted files (will be deleted when virt-v2v terminates,
* converted VM.
This is not a good idea for large VMs that have hunderds of GBs or even
TBs in size. The requirements on the necessary storage space can be
lessened with proper partitioning. I.e. source OVA and converted VM
don't end up on the same drive and TMPDIR is set to put even temporary
files somewhere else. But this is not a general solution. And sometimes
the necessary space may not be available at all.
The question is how to change the import path so that virt-v2v doesn't
have to extract the OVA. I can see the following solutions:
1) Solve it virt-v2v: create a layer for directly accessing the files
in the archive.
2) Solve it in QEMU: create backing method that would allow creating
qemu disk backed by the archive.
As long as the tar file is not compressed, accessing a file within it
should be trivial.
I asked Kevin if there is a way to get qemu to access a disk image at
an offset within another file, but there is no such feature at the
moment. It's possible with `losetup', but that requires root :-(
(At this point I would normally grumble about how easy this would be
with a microkernel, but I won't do that now.)
David Gilbert suggested looking at qemu-nbd which has an --offset
option, allowing a particular offset with another file to be accessed.
If we wanted to do it entirely within virt-v2v, I think this would be
the way to go - the complex logic could be hidden inside v2v/input_ova.ml
The second problem is to work out the right offset to use. I suspect
this is something that http://www.libarchive.org/ can do, and that
package is also in RHEL.
We could even imagine a qemu block backend based on libarchive.
3) Solve it on oVirt side: use some FUSE-based tool to provide
access to the archive and pass the OVA to virt-v2v not as a file but
as directory.
http://www.cybernoia.de/software/archivemount/ is one such tool which
can do this. It's not in RHEL, but it seems to be based on
libarchive.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
7:08 a.m.
On Fri, Sep 09, 2016 at 01:03:49PM +0200, Tomáš Golembiovský wrote:
Hi,
recently we (oVirt) have started discussing whether the way virt-v2v
handles import from OVA files is good. And I would be interested in
ideas how it can be improved. It is likely somebody already gave some
thought to this problem.
TL;DR: Extracting the OVA before import is a problem for large VMs (in
sizes of TBs). Can we change something to prevent the extraction and
work directly over OVA?
What we consider a huge shortcoming is the fact that whole OVA is
extracted prior to the import into a temporary directory and processed
afterwards. Under normal situation user can have up to three copies of
the VM on his drive at the end of import:
* original OVA,
* temporary extracted files (will be deleted when virt-v2v terminates,
* converted VM.
This is not a good idea for large VMs that have hunderds of GBs or even
TBs in size. The requirements on the necessary storage space can be
lessened with proper partitioning. I.e. source OVA and converted VM
don't end up on the same drive and TMPDIR is set to put even temporary
files somewhere else. But this is not a general solution. And sometimes
the necessary space may not be available at all.
The question is how to change the import path so that virt-v2v doesn't
have to extract the OVA. I can see the following solutions:
1) Solve it virt-v2v: create a layer for directly accessing the files
in the archive.
2) Solve it in QEMU: create backing method that would allow creating
qemu disk backed by the archive.
3) Solve it on oVirt side: use some FUSE-based tool to provide
access to the archive and pass the OVA to virt-v2v not as a file but
as directory.
Does anyone have any other ideas or suggestions?
Consider using virt-v2v --in-place, and doing your own import in
whatever way is most appropriate for your scenarios.
This is what we do in Virtuozzo to import VMs created in the previous
version (based on proprietary hypervisor) into Virtuozzo 7
(QEMU/KVM-based).
More specifically,
1) the target VM configuration is created from the source VM
configuration by Virtuozzo code
2) we have two scenarios for hdd images:
a) with data copying from a host running Virtuozzo 6 to a host
running Virtuozzo 7.
In this case the new qcow2 images are created and the data is
migrated by Virtuozzo code; then they are attached directly to the
new VM which is safe as the original images remain on the source
host. The new images are then modified during v2v --in-place and
then used by the new VM.
b) without data copying (e.g. if shared storage is used).
In this case qcow2 overlays are created with original images as
backing files, and attached to the new VM. Once v2v --in-place is
complete and the new VM starts its new life, the original images
can be lazily merged into the new qcow2 files or just left as is.
I think something like this can be done for your usecase, e.g. you can
create qcow2 images with images from the archive as backing files. One
possibility to do so is to have a loopback block device on top of the
archive with appropriate offset and length (OVA are just tarballs,
aren't they? So every archive member is just a contiguous chunk of data
within the file.) Upon completing the conversion and starting the
imported VM you can merge all the data into the new images and get rid
of the source tarball.
Roman.
7:18 a.m.
On Fri, Sep 09, 2016 at 03:08:31PM +0300, Roman Kagan wrote:
On Fri, Sep 09, 2016 at 01:03:49PM +0200, Tomáš Golembiovský wrote:
I think something like this can be done for your usecase, e.g. you can
create qcow2 images with images from the archive as backing files. One
possibility to do so is to have a loopback block device on top of the
archive with appropriate offset and length
Alternatively, you can make one layer less if you teach QEMU raw block
driver to recongnize offset and length options.
Roman.
7:32 a.m.
On 09 Sep 2016, at 14:02, Richard W.M. Jones
<rjones(a)redhat.com> wrote:
On Fri, Sep 09, 2016 at 01:03:49PM +0200, Tomáš Golembiovský wrote:
> Hi,
>
> recently we (oVirt) have started discussing whether the way virt-v2v
> handles import from OVA files is good. And I would be interested in
> ideas how it can be improved. It is likely somebody already gave some
> thought to this problem.
>
> TL;DR: Extracting the OVA before import is a problem for large VMs (in
> sizes of TBs). Can we change something to prevent the extraction and
> work directly over OVA?
Specifically virt-v2v needs to do:
qemu-img create -b <source-file-within-the-tarball> -f qcow2 overlay.qcow2
qemu-img convert overlay.qcow2 output
> What we consider a huge shortcoming is the fact that whole OVA is
> extracted prior to the import into a temporary directory and processed
> afterwards. Under normal situation user can have up to three copies of
> the VM on his drive at the end of import:
>
> * original OVA,
> * temporary extracted files (will be deleted when virt-v2v terminates,
> * converted VM.
>
>
> This is not a good idea for large VMs that have hunderds of GBs or even
> TBs in size. The requirements on the necessary storage space can be
> lessened with proper partitioning. I.e. source OVA and converted VM
> don't end up on the same drive and TMPDIR is set to put even temporary
> files somewhere else. But this is not a general solution. And sometimes
> the necessary space may not be available at all.
>
>
> The question is how to change the import path so that virt-v2v doesn't
> have to extract the OVA. I can see the following solutions:
>
> 1) Solve it virt-v2v: create a layer for directly accessing the files
> in the archive.
>
> 2) Solve it in QEMU: create backing method that would allow creating
> qemu disk backed by the archive.
As long as the tar file is not compressed, accessing a file within it
should be trivial.
The OVA standard [1] talks about compression. But it looks like it’s meant only for
individual disks inside the archive. It doesn’t seem to be clear about it.
The OVF xml is guaranteed to be at the beginning so it doesn’t need to read a lot until
really reading it whole even if it would be compressed.
Well, I guess it’s a reasonable to start with plain tar regardless.
Thanks,
michal
[1] http://www.dmtf.org/sites/default/files/standards/documents/DSP0243_2.1.1...
I asked Kevin if there is a way to get qemu to access a disk image at
an offset within another file, but there is no such feature at the
moment. It's possible with `losetup', but that requires root :-(
(At this point I would normally grumble about how easy this would be
with a microkernel, but I won't do that now.)
David Gilbert suggested looking at qemu-nbd which has an --offset
option, allowing a particular offset with another file to be accessed.
If we wanted to do it entirely within virt-v2v, I think this would be
the way to go - the complex logic could be hidden inside v2v/input_ova.ml
The second problem is to work out the right offset to use. I suspect
this is something that http://www.libarchive.org/ can do, and that
package is also in RHEL.
We could even imagine a qemu block backend based on libarchive.
> 3) Solve it on oVirt side: use some FUSE-based tool to provide
> access to the archive and pass the OVA to virt-v2v not as a file but
> as directory.
http://www.cybernoia.de/software/archivemount/ is one such tool which
can do this. It's not in RHEL, but it seems to be based on
libarchive.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
7:49 a.m.
Hi,
just to keep this thread alive, here are few comments on the progress.
(Mostly for those not following QEMU mailing lists.)
On Fri, 9 Sep 2016 15:18:02 +0300
Roman Kagan <rkagan(a)virtuozzo.com> wrote:
On Fri, Sep 09, 2016 at 03:08:31PM +0300, Roman Kagan wrote:
> On Fri, Sep 09, 2016 at 01:03:49PM +0200, Tomáš Golembiovský wrote:
> I think something like this can be done for your usecase, e.g. you can
> create qcow2 images with images from the archive as backing files. One
> possibility to do so is to have a loopback block device on top of the
> archive with appropriate offset and length
Using the loopback device would be the easiest solution in the sense
that it wouldn't require code changes outside virt-v2v. But, there's a
big issue: it requires root privileges. That's why I cast this idea away
from the beginning and decided to use that only as the last resort.
Other suggestion was to use qemu-nbd because it already has --offset
argument to set where the disk starts. It turned out that:
a) The code for --offset is buggy and clients may attempt to read data
outside the backing file, because qemu-nbd reports wrong size of the
disk. Such request will inevitably fail.
b) We also need to be able to specify the size of the disk. It cannot
be left undefined and limited only by the length of underlying file.
Security reasons aside, there are also some practical reasons. E.g.
it turned out that VMDK format stores some footer at the end of the
disk. If the size of the disk is not set properly the VMDK driver
fails to find the footer and regards the disk as invalid.
I fixed the issues and sent patches to QEMU [1][2].
Quickly somebody commented [3] that doing this in qemu-nbd is not the
right idea and it should be done in the raw driver...
Alternatively, you can make one layer less if you teach QEMU raw
block
driver to recongnize offset and length options.
Roman.
... that is something what Roman suggested from the start.
It'll be more complicated than fixing qemu-nbd, but I gave it a shot. I
have sent the first version [4] of that to QEMU. I would expect it going
in right away, there's still some work to be done.
That being said, I also have POC ready (attached) for virt-v2v. It's
rather dirty as it contains some 'tar' invocations to get the necessary
information from the archive. There are some OCAML modules for working
with tar archives, but I didn't try any yet.
Good thing is that QEMU people are not against putting the size & offset
parameters into raw driver. Still it may turn out to be a task too hard.
In that case, as a fallback, we can still use the NBD approach. We don't
have to use qemu-nbd. We could also try to submit code for size & offset
into the nbd-server in NBD tools (the defacto reference implementation).
But that's something I didn't pursue yet.
Tomas
[1] https://lists.nongnu.org/archive/html/qemu-devel/2016-09/msg04554.html
[2] https://lists.nongnu.org/archive/html/qemu-devel/2016-09/msg04556.html
[3] https://lists.nongnu.org/archive/html/qemu-devel/2016-09/msg04578.html
[4] https://lists.nongnu.org/archive/html/qemu-block/2016-10/msg00008.html
--
Tomáš Golembiovský <tgolembi(a)redhat.com>
2853
days inactive
2877
days old
6 comments
4 participants
participants (4)
-
Michal Skrivanek -
Richard W.M. Jones -
Roman Kagan -
Tomáš Golembiovský