4:29 a.m.
For a long time I've wanted to split up virt-v2v into smaller
components to make it easier to consume. It's never been clear how to
do this, but I think I have a workable plan now, described in this email.
----------------------------------------------------------------------
First, the AIMS, which are:
(a) Preserve current functionality, including copying conversion,
in-place conversion, and the virt-v2v command line.
(b) Allow warm migration to use virt-v2v without requiring the
"--debug-overlays hack".
(c) Allow threads, multi-conn, and parallel copying of guest disks, all
for better copying performance.
(d) Allow an alternate supervisor to convert and copy many guests in
parallel, given that the supervisor has a global view of the
system/network (I'm not intending to implement this, only to make
it possible).
(e) Better progress bars.
(f) Better logging.
(g) Reuse as much existing code as possible. This is NOT a rewrite!
----------------------------------------------------------------------
Here's my PLAN:
/usr/bin/virt-v2v still exists, but it's now a supervisor program
(possibly even a shell script) that runs the steps below:
(1) Set up the input side by running "helper-v2v-input-<type>". For
all input types this creates a temporary directory containing:
/tmp/XXXXXX/in1 NBD endpoints overlaying the source disk(s)
/tmp/XXXXXX/in2 (these are actually Unix domain sockets)
/tmp/XXXXXX/in3
/tmp/XXXXXX/metadata.in Metadata parsed from the source.
Currently for most inputs we have a running nbdkit process for
each source disk, and we'd do the same here, except we add
nbdkit-cow-filter on top so that the source disk is protected from
being modified. Another small difference is that for -i disk
(local input) we would need an active nbdkit process on top of the
disk, whereas currently we set the disk as a qcow2 backing file.
(2) Perform the conversion by running "helper-v2v-convert". This does
the conversion and sparsification. It writes directly to the NBD
endpoints (in*) above. The writes are stored in the COW overlay
so the source disk is not modified.
Conversion will also create an output metadata file:
/tmp/XXXXXX/metadata.out Target metadata
Exact format of the metadata files is to be decided, but some kind
of not-quite-libvirt-XML may be suitable. It's also not clear if
the metadata format is an internal detail of virt-v2v, or if we
document it as a stable API.
(3) Set up the output side by running "helper-v2v-output-<type>
setup". This will read the output metadata and do whatever is
needed to set up the empty output disks (perhaps by creating a
guest on the target, but also this could be done in step (5)
below).
This will create:
/tmp/XXXXXX/out1 NBD endpoints overlaying the target disk(s)
/tmp/XXXXXX/out2 (these are actually Unix domain sockets)
/tmp/XXXXXX/out3
(4) Do the copy. By default this will run either nbdcopy or qemu-img
convert from in* -> out*.
Copying could be done in parallel, currently it is done serially.
(5) Finalize the output by running "helper-v2v-output-<type> final".
This might create the target guest and whatever else is needed.
(6) Kill the NBD servers and clean up the temporary directory.
----------------------------------------------------------------------
Let's see how this plan matches the aims.
Aim (a):
Copying conversion works as outlined above. In-place conversion
works by placing an NBD server on top of the files you want to
convert and running helper-v2v-convert (virt-v2v --in-place would
also still work for backwards compat).
Aim (b):
Warm migration: Should be fairly clear this can work in the same way
as in-place conversion, but I'll discuss this further with Martin K
and Tomas to make sure I'm not missing anything.
Aims (c), (d):
Threads etc for performance: Although I don't plan to implement
this, it's clear that an alternate supervisor program could improve
performance here by either doing copies of a single guest / multiple
disks in parallel, but even better by having a global view of the
system and doing copies of multiple guests' disks in parallel.
This is outside the scope of the virt-v2v project, but in scope for
something like MTV.
Aim (e):
Better progress bars: nbdcopy should have support for
machine-readable progress bars, once I push the changes. It will
mean no more need to parse debug logs.
Aim (f):
Better logging: I hope we can log each step separately.
A custom supervisor program would also be able to tell which
particular step failed (eg. did it fail in conversion? did it fail
copying a disk and which one?)
Aim (g):
This works by splitting up the existing v2v code base into separate
binaries. It is already broadly structured (internally) like this.
So it's not a rewrite, it's a big refactoring.
However I'd probably write a new virt-v2v supervisor binary, because
the existing command line parsing code is extremely complex.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
10:58 a.m.
On Wed, Nov 25, 2020 at 12:31 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
For a long time I've wanted to split up virt-v2v into smaller
components to make it easier to consume. It's never been clear how to
do this, but I think I have a workable plan now, described in this email.
----------------------------------------------------------------------
First, the AIMS, which are:
(a) Preserve current functionality, including copying conversion,
in-place conversion, and the virt-v2v command line.
(b) Allow warm migration to use virt-v2v without requiring the
"--debug-overlays hack".
(c) Allow threads, multi-conn, and parallel copying of guest disks, all
for better copying performance.
(d) Allow an alternate supervisor to convert and copy many guests in
parallel, given that the supervisor has a global view of the
system/network (I'm not intending to implement this, only to make
it possible).
(e) Better progress bars.
(f) Better logging.
(g) Reuse as much existing code as possible. This is NOT a rewrite!
Sounds good
----------------------------------------------------------------------
Here's my PLAN:
/usr/bin/virt-v2v still exists, but it's now a supervisor program
(possibly even a shell script) that runs the steps below:
(1) Set up the input side by running "helper-v2v-input-<type>". For
all input types this creates a temporary directory containing:
/tmp/XXXXXX/in1 NBD endpoints overlaying the source disk(s)
/tmp/XXXXXX/in2 (these are actually Unix domain sockets)
/tmp/XXXXXX/in3
/tmp/XXXXXX/metadata.in Metadata parsed from the source.
Currently for most inputs we have a running nbdkit process for
each source disk, and we'd do the same here, except we add
nbdkit-cow-filter on top so that the source disk is protected from
being modified. Another small difference is that for -i disk
(local input) we would need an active nbdkit process on top of the
disk, whereas currently we set the disk as a qcow2 backing file.
(2) Perform the conversion by running "helper-v2v-convert". This does
the conversion and sparsification. It writes directly to the NBD
endpoints (in*) above. The writes are stored in the COW overlay
so the source disk is not modified.
Conversion will also create an output metadata file:
/tmp/XXXXXX/metadata.out Target metadata
Exact format of the metadata files is to be decided, but some kind
of not-quite-libvirt-XML may be suitable. It's also not clear if
the metadata format is an internal detail of virt-v2v, or if we
document it as a stable API.
(3) Set up the output side by running "helper-v2v-output-<type>
setup". This will read the output metadata and do whatever is
needed to set up the empty output disks (perhaps by creating a
guest on the target, but also this could be done in step (5)
below).
This will create:
/tmp/XXXXXX/out1 NBD endpoints overlaying the target disk(s)
/tmp/XXXXXX/out2 (these are actually Unix domain sockets)
/tmp/XXXXXX/out3
For oVirt, this step can create the image transfer.
(4) Do the copy. By default this will run either nbdcopy or
qemu-img
convert from in* -> out*.
For oVirt, if we have a qcow2 image with a backing file connected to nbdkit
connected to vddk, we can upload the disk using imageio client,
already supporting multiple connections. For local disk (-i) it is about 2 times
faster compared with qemu-img convert + nbdkit + rhv-upload-plugin.
I think a generic implementation using nbdcopy or "qemu-img convert" is
a good idea, but specific output should be able to control the copy step.
Copying could be done in parallel, currently it is done
serially.
(5) Finalize the output by running "helper-v2v-output-<type> final".
This might create the target guest and whatever else is needed.
For oVirt we would finalize the transfer and create the VM here.
(6) Kill the NBD servers and clean up the temporary directory.
----------------------------------------------------------------------
Let's see how this plan matches the aims.
Aim (a):
Copying conversion works as outlined above. In-place conversion
works by placing an NBD server on top of the files you want to
convert and running helper-v2v-convert (virt-v2v --in-place would
also still work for backwards compat).
Aim (b):
Warm migration: Should be fairly clear this can work in the same way
as in-place conversion, but I'll discuss this further with Martin K
and Tomas to make sure I'm not missing anything.
Aims (c), (d):
Threads etc for performance: Although I don't plan to implement
this, it's clear that an alternate supervisor program could improve
performance here by either doing copies of a single guest / multiple
disks in parallel, but even better by having a global view of the
system and doing copies of multiple guests' disks in parallel.
This is outside the scope of the virt-v2v project, but in scope for
something like MTV.
Aim (e):
Better progress bars: nbdcopy should have support for
machine-readable progress bars, once I push the changes. It will
mean no more need to parse debug logs.
I don't think this is only about the progress bar, but the entire process.
The tool should have an option to generate output for machines, for example
GUI showing what the tool is doing. Typically such application needs to get
events when changing the state or when progress changes.
It can be simple stream of lines:
event: Setting up ...
event: Converting disk ...
event: Copying disk ...
progress: 0
progress: 1
...
progress: 100
event: Creating vm ...
The program consuming this can change the text on each "event:" line and
update the progress bar on each "progress:" line.
This can be enhanced by providing additional info about the process. For example
in RHV it will be helpful if we have the transfer id, disk id, and vm
id when try to
debug failures. Currently we have to use keep debug logs and grep the log to get
the values, but it would be more useful if the details were reported
by the tool:
info: disk_id=xxxyyy transfer_id=yyyzzz
If the process failed the tool can report:
error: Why it failed...
This is something that every tool doing complex/slow stuff needs to
implement, like
qemu-img convert or dd.
Like https://testanything.org/ for watching processes - watch anything protocol?
Aim (f):
Better logging: I hope we can log each step separately.
A custom supervisor program would also be able to tell which
particular step failed (eg. did it fail in conversion? did it fail
copying a disk and which one?)
For logging it would be useful to support logging to file, so tools
do not have to read the logs and write them to file.
This is true for all virt-* tools.
Aim (g):
This works by splitting up the existing v2v code base into separate
binaries. It is already broadly structured (internally) like this.
So it's not a rewrite, it's a big refactoring.
However I'd probably write a new virt-v2v supervisor binary, because
the existing command line parsing code is extremely complex.
Nir
2:12 a.m.
On Mon, Nov 30, 2020 at 06:58:00PM +0200, Nir Soffer wrote:
On Wed, Nov 25, 2020 at 12:31 PM Richard W.M. Jones
<rjones(a)redhat.com> wrote> > Aim (e):
>
> Better progress bars: nbdcopy should have support for
> machine-readable progress bars, once I push the changes. It will
> mean no more need to parse debug logs.
I don't think this is only about the progress bar, but the entire process.
The tool should have an option to generate output for machines, for example
GUI showing what the tool is doing. Typically such application needs to get
events when changing the state or when progress changes.
It can be simple stream of lines:
event: Setting up ...
event: Converting disk ...
event: Copying disk ...
progress: 0
progress: 1
...
progress: 100
event: Creating vm ...
The program consuming this can change the text on each "event:" line and
update the progress bar on each "progress:" line.
I suspect the reality will be that these will have to go to several
separate file descriptors, just because that's the only way to make it
work when you have separate processes doing to work. However it
should be an improvement over trying to parse debug logs.
This can be enhanced by providing additional info about the
process. For example in RHV it will be helpful if we have the
transfer id, disk id, and vm id when try to debug
failures. Currently we have to use keep debug logs and grep the log
to get the values, but it would be more useful if the details were
reported by the tool:
info: disk_id=xxxyyy transfer_id=yyyzzz
If the process failed the tool can report:
error: Why it failed...
This is something that every tool doing complex/slow stuff needs to
implement, like qemu-img convert or dd.
Like https://testanything.org/ for watching processes - watch anything protocol?
OK.
> Aim (f):
>
> Better logging: I hope we can log each step separately.
>
> A custom supervisor program would also be able to tell which
> particular step failed (eg. did it fail in conversion? did it fail
> copying a disk and which one?)
For logging it would be useful to support logging to file, so tools
do not have to read the logs and write them to file.
This is true for all virt-* tools.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
5:24 a.m.
On Wed, Nov 25, 2020 at 10:29:45AM +0000, Richard W.M. Jones wrote:
For a long time I've wanted to split up virt-v2v into smaller
components to make it easier to consume. It's never been clear how to
do this, but I think I have a workable plan now, described in this email.
In contrast I am replying for a long time here as well. I know we actually
talked about all this on IRC, but I them planned to post the summary here and I
did not. Unfortunately I cannot find this in my IRC logs any more, so I will
try to remember as many things as possible.
----------------------------------------------------------------------
First, the AIMS, which are:
(a) Preserve current functionality, including copying conversion,
in-place conversion, and the virt-v2v command line.
(b) Allow warm migration to use virt-v2v without requiring the
"--debug-overlays hack".
(c) Allow threads, multi-conn, and parallel copying of guest disks, all
for better copying performance.
(d) Allow an alternate supervisor to convert and copy many guests in
parallel, given that the supervisor has a global view of the
system/network (I'm not intending to implement this, only to make
it possible).
(e) Better progress bars.
(f) Better logging.
(g) Reuse as much existing code as possible. This is NOT a rewrite!
So my idea was that this could be split into phases similar to what is shown in
v2v/types.mli, separately per each input and output object and then some core
functionality. Any internal state could be kept in a specific file so that it
is accessible to all these helpers. After that virt-v2v _could_ be implemented
by a shell script if someone wanted. Of course that is just for illustration of
how usable that would be, not that it would be viable to make that a shell
script, of course.
There are some command line options to limit virt-v2v to do only some phase or
skip a phase (--print-source, --no-copy, -o null) and this would make it more
approachable. Serialising the internal state into a file is something that does
not have to be parsed by anyone else and can also be pretty relaxed when it
comes to backward compatibility (two input helpers will probably not need to be
run from two different versions of virt-v2v).
What would be nice to have exposed is the internal representation of all the
information needed to construct an output guest description. I would have to be
in an extensible format, but it would not be prone to getting stale as often.
Think of it as `-o json` and then few helpers that convert this type of
information into the output format. What would be nice about this is that
supporting any new format would not require a new output type for each tiny
change. Keeping some existing format up to date could also be easier because
the clear split could lower the barrier for developers to support their cloud
solution format. Of course I do not know how much of an issue this currently is
(or is not), but it seemed like a good idea to me.
----------------------------------------------------------------------
Here's my PLAN:
/usr/bin/virt-v2v still exists, but it's now a supervisor program
(possibly even a shell script) that runs the steps below:
(1) Set up the input side by running "helper-v2v-input-<type>". For
all input types this creates a temporary directory containing:
/tmp/XXXXXX/in1 NBD endpoints overlaying the source disk(s)
/tmp/XXXXXX/in2 (these are actually Unix domain sockets)
/tmp/XXXXXX/in3
/tmp/XXXXXX/metadata.in Metadata parsed from the source.
Currently for most inputs we have a running nbdkit process for
each source disk, and we'd do the same here, except we add
nbdkit-cow-filter on top so that the source disk is protected from
being modified. Another small difference is that for -i disk
(local input) we would need an active nbdkit process on top of the
disk, whereas currently we set the disk as a qcow2 backing file.
(2) Perform the conversion by running "helper-v2v-convert". This does
the conversion and sparsification. It writes directly to the NBD
endpoints (in*) above. The writes are stored in the COW overlay
so the source disk is not modified.
This would make it easy for someone to copy the disks themselves and then
provide them as nbd sockets if they want to modify the copies in place, locally,
*after* copying and without any extra cow layer on top. That is good.
Conversion will also create an output metadata file:
/tmp/XXXXXX/metadata.out Target metadata
Exact format of the metadata files is to be decided, but some kind
of not-quite-libvirt-XML may be suitable. It's also not clear if
the metadata format is an internal detail of virt-v2v, or if we
document it as a stable API.
(3) Set up the output side by running "helper-v2v-output-<type>
setup". This will read the output metadata and do whatever is
needed to set up the empty output disks (perhaps by creating a
guest on the target, but also this could be done in step (5)
below).
This will create:
/tmp/XXXXXX/out1 NBD endpoints overlaying the target disk(s)
/tmp/XXXXXX/out2 (these are actually Unix domain sockets)
/tmp/XXXXXX/out3
(4) Do the copy. By default this will run either nbdcopy or qemu-img
convert from in* -> out*.
Copying could be done in parallel, currently it is done serially.
(5) Finalize the output by running "helper-v2v-output-<type> final".
This might create the target guest and whatever else is needed.
(6) Kill the NBD servers and clean up the temporary directory.
Of course the suggested split is not required, what you suggest here would work
just as well. I just wanted share the idea I had because I thought it could
actually be easier to do, maintain, and future-proof.
----------------------------------------------------------------------
Let's see how this plan matches the aims.
Aim (a):
Copying conversion works as outlined above. In-place conversion
works by placing an NBD server on top of the files you want to
convert and running helper-v2v-convert (virt-v2v --in-place would
also still work for backwards compat).
I remember --in-place doing some input-related shenanigans that made it
different from "just convert this". But I think keeping the original
--in-place
will not cause any issues.
Aim (b):
Warm migration: Should be fairly clear this can work in the same way
as in-place conversion, but I'll discuss this further with Martin K
and Tomas to make sure I'm not missing anything.
The separation of steps works a bit better. I think keeping the pre-checks and
everything is good, it's just that when one is messing up with the workflow it
is easier to plug various phases together at different times when it is more
split apart. I would imagine high-level debugging by non-expert is easier as
well.
Aims (c), (d):
Threads etc for performance: Although I don't plan to implement
this, it's clear that an alternate supervisor program could improve
performance here by either doing copies of a single guest / multiple
disks in parallel, but even better by having a global view of the
system and doing copies of multiple guests' disks in parallel.
This is outside the scope of the virt-v2v project, but in scope for
something like MTV.
And easy to do with the split ;-)
Aim (e):
Better progress bars: nbdcopy should have support for
machine-readable progress bars, once I push the changes. It will
mean no more need to parse debug logs.
Aim (f):
Better logging: I hope we can log each step separately.
A custom supervisor program would also be able to tell which
particular step failed (eg. did it fail in conversion? did it fail
copying a disk and which one?)
Aim (g):
This works by splitting up the existing v2v code base into separate
binaries. It is already broadly structured (internally) like this.
So it's not a rewrite, it's a big refactoring.
However I'd probably write a new virt-v2v supervisor binary, because
the existing command line parsing code is extremely complex.
Sounds similar to what I thought. And if it is simplified, then virt-v2v can
just forward arguments to appropriate places. Sounds good.
I do not know if I mentioned everything and I am not sure how deep we went into
some of the details, but I guess the best way will show up later on no matter
what.
Once again sorry for such a late public reply, hopefully this will at least keep
part of the conversation archived =)
Have a great day,
Martin
1376
days inactive
1488
days old
3 comments
3 participants
participants (3)
-
Martin Kletzander -
Nir Soffer -
Richard W.M. Jones