4:44 a.m.
So it didn't make any noticable difference in my test. I wonder if
the test guest I'm using (Fedora 32 using dracut) doesn't use parallel
compression?
However I don't think it can cause a problem and it seems obvious that
it could benefit some cases.
Rich.
4:44 a.m.
This helps mkinitrd which runs pigz (parallel gzip).
Thanks Jean-Louis Dupond for suggesting this change.
---
common | 2 +-
v2v/v2v.ml | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/common b/common
index c840f2e39..ea5278bba 160000
--- a/common
+++ b/common
@@ -1 +1 @@
-Subproject commit c840f2e39d0bb637a98b224c89f6df011e1d4414
+Subproject commit ea5278bba7800e3a6d9c233a0f5f53a9acf244f8
diff --git a/v2v/v2v.ml b/v2v/v2v.ml
index 73edff2c4..7637b1a8c 100644
--- a/v2v/v2v.ml
+++ b/v2v/v2v.ml
@@ -88,6 +88,8 @@ let rec main () =
let g = open_guestfs ~identifier:"v2v" () in
g#set_memsize (g#get_memsize () * 14 / 5);
+ (* Setting the number of vCPUs allows parallel mkinitrd. *)
+ g#set_smp (Sysconf.nr_processors_online ());
(* The network is only used by the unconfigure_vmware () function. *)
g#set_network true;
(match conversion_mode with
--
2.27.0
7:03 a.m.
On Friday, 18 September 2020 11:44:04 CEST Richard W.M. Jones wrote:
let g = open_guestfs ~identifier:"v2v" () in
g#set_memsize (g#get_memsize () * 14 / 5);
+ (* Setting the number of vCPUs allows parallel mkinitrd. *)
+ g#set_smp (Sysconf.nr_processors_online ());
IMHO this is not a good idea, for few reasons:
a) it unconditionally uses all the available CPUs with no way to change
it to any other value
b) virt-v2v is run also on systems that are not specifically dedicated
as conversion servers, for example the oVirt hosts (that run the oVirt
VMs); in this context, suddently using more CPUs than requested may
saturate the system resources
IMHO the right approach is to add a --smp N option like available in
other libguestfs tools; this way, the user can decide how many
resources virt-v2v can use.
--
Pino Toscano
7:15 a.m.
On Fri, Sep 18, 2020 at 02:03:00PM +0200, Pino Toscano wrote:
On Friday, 18 September 2020 11:44:04 CEST Richard W.M. Jones wrote:
> let g = open_guestfs ~identifier:"v2v" () in
> g#set_memsize (g#get_memsize () * 14 / 5);
> + (* Setting the number of vCPUs allows parallel mkinitrd. *)
> + g#set_smp (Sysconf.nr_processors_online ());
IMHO this is not a good idea, for few reasons:
a) it unconditionally uses all the available CPUs with no way to change
it to any other value
My second suggestion would be to use:
+ g#set_smp (min 8 (Sysconf.nr_processors_online ()));
(limit to be discussed, but here I arbitrarily chose 8). I agree that
it's not a good idea to use all CPUs. However:
b) virt-v2v is run also on systems that are not specifically
dedicated
as conversion servers, for example the oVirt hosts (that run the oVirt
VMs); in this context, suddently using more CPUs than requested may
saturate the system resources
It's fairly unlikely this would happen. All the libguestfs steps are
still serialized. Only a few steps would use all CPUs, including
dracut/mkinitrd, and possibly some RPM commands.
IMHO the right approach is to add a --smp N option like available in
other libguestfs tools; this way, the user can decide how many
resources virt-v2v can use.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
4:52 a.m.
On Fri, Sep 18, 2020 at 10:44:03AM +0100, Richard W.M. Jones wrote:
So it didn't make any noticable difference in my test. I wonder
if
the test guest I'm using (Fedora 32 using dracut) doesn't use parallel
compression?
Do you do anything special to optimize storage ? If the thing using
parallel CPUs in the guest is doing I/O you'd likely want to tune
storage at same time. In current QEMU git master there's a change[1]
that sets viort-blk/virtio-scsi num_queues to match SMP count,
so the guest OS get fully parallel I/O from meach guest CPU.
[1] a4eef0711b2cf7a7476c3e2c202a414b68a1baa0
However I don't think it can cause a problem and it seems obvious
that
it could benefit some cases.
What's the default RAM size ? On hosts with very large numbers of CPUs
(100's) you might need to scale up RAM size too so you keep a sane
RAM-per-CPU ratio. Or default to host CPUs, but cap it at say 32.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
5:22 a.m.
On Fri, Sep 18, 2020 at 10:52:58AM +0100, Daniel P. Berrangé wrote:
On Fri, Sep 18, 2020 at 10:44:03AM +0100, Richard W.M. Jones wrote:
> So it didn't make any noticable difference in my test. I wonder if
> the test guest I'm using (Fedora 32 using dracut) doesn't use parallel
> compression?
Do you do anything special to optimize storage ? If the thing using
parallel CPUs in the guest is doing I/O you'd likely want to tune
storage at same time. In current QEMU git master there's a change[1]
that sets viort-blk/virtio-scsi num_queues to match SMP count,
so the guest OS get fully parallel I/O from meach guest CPU.
[1] a4eef0711b2cf7a7476c3e2c202a414b68a1baa0
I'm not using a new enough qemu to have that and it doesn't appear in
a released version, but I'll test again once we have a new enough
version.
> However I don't think it can cause a problem and it seems
obvious that
> it could benefit some cases.
What's the default RAM size ? On hosts with very large numbers of CPUs
(100's) you might need to scale up RAM size too so you keep a sane
RAM-per-CPU ratio. Or default to host CPUs, but cap it at say 32.
I'll cap the number of CPUs in this patch.
Thanks,
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
5:26 a.m.
On Fri, Sep 18, 2020 at 11:22:15AM +0100, Richard W.M. Jones wrote:
On Fri, Sep 18, 2020 at 10:52:58AM +0100, Daniel P. Berrangé wrote:
> On Fri, Sep 18, 2020 at 10:44:03AM +0100, Richard W.M. Jones wrote:
> > So it didn't make any noticable difference in my test. I wonder if
> > the test guest I'm using (Fedora 32 using dracut) doesn't use parallel
> > compression?
>
> Do you do anything special to optimize storage ? If the thing using
> parallel CPUs in the guest is doing I/O you'd likely want to tune
> storage at same time. In current QEMU git master there's a change[1]
> that sets viort-blk/virtio-scsi num_queues to match SMP count,
> so the guest OS get fully parallel I/O from meach guest CPU.
>
> [1] a4eef0711b2cf7a7476c3e2c202a414b68a1baa0
I'm not using a new enough qemu to have that and it doesn't appear in
a released version, but I'll test again once we have a new enough
version.
You could set num_queues explicitly in v2v if desired, as the actual
config knob has been around a while, only the default is changing in
QEMU.
> > However I don't think it can cause a problem and it
seems obvious that
> > it could benefit some cases.
>
> What's the default RAM size ? On hosts with very large numbers of CPUs
> (100's) you might need to scale up RAM size too so you keep a sane
> RAM-per-CPU ratio. Or default to host CPUs, but cap it at say 32.
I'll cap the number of CPUs in this patch.
Thanks,
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1450
days inactive
1450
days old
6 comments
3 participants
participants (3)
-
Daniel P. Berrangé -
Pino Toscano -
Richard W.M. Jones