1:04 p.m.
Hi Abhay,
FYI I thought I would document the successful commands I am using to
benchmark nbdcopy and produce the flame graphs that you saw this
morning. Attached is a very recent flame graph produced using this
method.
Firstly I'm running everything on Fedora 34, with selected packages
upgraded to Fedora Rawhide. However any reasonably recent Linux
distro should work fine. You will need to install the perf tool.
Compile libnbd & nbdkit from git source, following the instructions in
the respective README files.
https://gitlab.com/nbdkit/libnbd
https://gitlab.com/nbdkit/nbdkit
I have nbdkit and libnbd checked out in adjacent directories. This is
important so that commands like "./nbdkit" and "../libnbd/run
nbdcopy"
work. There's more information about this in the READMEs.
I ran perf as below. Although nbdcopy and nbdkit themselves do not
require root (and usually should _not_ be run as root), in this case
perf must be run as root, so everything has to be run as root.
# perf record -a -g --call-graph=dwarf ./nbdkit -U - sparse-random size=1T --run
"MALLOC_CHECK_= ../libnbd/run nbdcopy \$uri \$uri"
Some things to explain:
* The output is perf.data in the local directory. This file may be
huge (22GB for me!)
* I am running this from the nbdkit directory, so ./nbdkit runs the
locally compiled copy of nbdkit. This allows me to make quick
changes to nbdkit and see the effects immediately.
* I am running nbdcopy using "../libnbd/run nbdcopy", so that's from
the adjacent locally compiled libnbd directory. Again the reason
for this is so I can make changes, recompile libnbd, and see the
effect quickly.
* "MALLOC_CHECK_=" is needed because of complicated reasons to do
with how the nbdkit wrapper enables malloc-checking. We should
probably provide a way to disable malloc-checking when benchmarking
because it adds overhead for no benefit, but I've not done that yet
(patches welcome!)
* The test harness is nbdkit-sparse-random-plugin, documented here:
https://libguestfs.org/nbdkit-sparse-random-plugin.1.html
* I'm using DWARF debugging info to generate call stacks, which is
more reliable than the default (frame pointers).
* The -a option means I'm measuring events on the whole machine. You
can read the perf manual to find out how to measure only a single
process (eg. just nbdkit or just nbdcopy). But actually measuring
the whole machine gives a truer picture, I believe.
* If the test takes too long to run or runs out of space, try
adjusting the size (1T = 1 terabyte) downwards, eg. 512G, 256G, ...
until it fits. Although nbdkit doesn't store the virtual disk or
use very much memory at all, the test does appear to stress the
Linux VMM, and the amount of perf.data generated can be huge.
Then I run this long command to generate the flame graph. Again
it must be run as root:
# perf script | ../FlameGraph/stackcollapse-perf.pl | ../FlameGraph/flamegraph.pl >
nbdcopy.svg
* This reads perf.data as input.
* Brendan Gregg's FlameGraph code is checked out in another adjacent
directory.
You can open the SVG file in a web browser. Try clicking around -
it's interactive.
If you get stuck, ask questions, we're here to help.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
1:18 p.m.
Thanks I was able to get perf.data for and can analyse it using perf report
which showed expected output but flamegraph errors for quite some time
Turns out I had to use stackcollapse-perf.pl instead of stackcollapse.pl
thanks!!
3:40 a.m.
On Tue, May 25, 2021 at 9:06 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
I ran perf as below. Although nbdcopy and nbdkit themselves do not
require root (and usually should _not_ be run as root), in this case
perf must be run as root, so everything has to be run as root.
# perf record -a -g --call-graph=dwarf ./nbdkit -U - sparse-random size=1T --run
"MALLOC_CHECK_= ../libnbd/run nbdcopy \$uri \$uri"
This uses 64 requests with a request size of 32m. In my tests using
--requests 16 --request-size 1048576 is faster. Did you try to profile
this?
Some things to explain:
* The output is perf.data in the local directory. This file may be
huge (22GB for me!)
* I am running this from the nbdkit directory, so ./nbdkit runs the
locally compiled copy of nbdkit. This allows me to make quick
changes to nbdkit and see the effects immediately.
* I am running nbdcopy using "../libnbd/run nbdcopy", so that's from
the adjacent locally compiled libnbd directory. Again the reason
for this is so I can make changes, recompile libnbd, and see the
effect quickly.
* "MALLOC_CHECK_=" is needed because of complicated reasons to do
with how the nbdkit wrapper enables malloc-checking. We should
probably provide a way to disable malloc-checking when benchmarking
because it adds overhead for no benefit, but I've not done that yet
(patches welcome!)
Why enable malloc checking in nbdkit when profiling nbdcopy?
* The test harness is nbdkit-sparse-random-plugin, documented here:
https://libguestfs.org/nbdkit-sparse-random-plugin.1.html
Does it create a similar pattern to real world images, or more like
the worst case?
In my tests using nbdkit memory and pattern plugins was way more
stable compared with real images via qemu-nbd/nbdkit, but real image
give more real results :-)
Maybe we can extract the extents from a real image, and add a plugin
accepting json extents and inventing data for the data extents?
* I'm using DWARF debugging info to generate call stacks, which
is
more reliable than the default (frame pointers).
When I tried to use perf, I did not get proper call stacks, maybe this
was the reason.
* The -a option means I'm measuring events on the whole machine.
You
can read the perf manual to find out how to measure only a single
process (eg. just nbdkit or just nbdcopy). But actually measuring
the whole machine gives a truer picture, I believe.
Why profile the whole machine? I would profile only nbdcopy or nbdkit,
depending on what we are trying to focus on.
Looking in the attached flame graph, if we focus on the nbdcopy worker_thread,
and sort by time:
poll_both_ends: 14.53% (58%)
malloc: 5.55% (22%)
nbd_ops_async_read: 4.34% (17%)
nbd_ops_get_extents: 0.52% (2%)
If we focus into into poll_both_ends:
send 10.17% (69%)
free 4.53% (31%)
So we have a lot of opportunities to optimize by allocating all buffers up front
as done in examples/libev-copy. But I'm not sure we will would see the same
picture when using smaller buffers (--request-size 1m).
nbd_ops_async_read is surprising - this is an async operation that should
consume no time. Why does it take 17% of the time?
Thanks for the info!
Nir
4:32 a.m.
On Wed, May 26, 2021 at 11:40:11AM +0300, Nir Soffer wrote:
On Tue, May 25, 2021 at 9:06 PM Richard W.M. Jones
<rjones(a)redhat.com> wrote:
> I ran perf as below. Although nbdcopy and nbdkit themselves do not
> require root (and usually should _not_ be run as root), in this case
> perf must be run as root, so everything has to be run as root.
>
> # perf record -a -g --call-graph=dwarf ./nbdkit -U - sparse-random size=1T --run
"MALLOC_CHECK_= ../libnbd/run nbdcopy \$uri \$uri"
This uses 64 requests with a request size of 32m. In my tests using
--requests 16 --request-size 1048576 is faster. Did you try to profile
this?
Interesting! No I didn't. In fact I just assumed that larger request
sizes / number of parallel requests would be better.
This would be an interesting and easy avenue for Abhay to explore too.
> Some things to explain:
>
> * The output is perf.data in the local directory. This file may be
> huge (22GB for me!)
>
> * I am running this from the nbdkit directory, so ./nbdkit runs the
> locally compiled copy of nbdkit. This allows me to make quick
> changes to nbdkit and see the effects immediately.
>
> * I am running nbdcopy using "../libnbd/run nbdcopy", so that's from
> the adjacent locally compiled libnbd directory. Again the reason
> for this is so I can make changes, recompile libnbd, and see the
> effect quickly.
>
> * "MALLOC_CHECK_=" is needed because of complicated reasons to do
> with how the nbdkit wrapper enables malloc-checking. We should
> probably provide a way to disable malloc-checking when benchmarking
> because it adds overhead for no benefit, but I've not done that yet
> (patches welcome!)
Why enable malloc checking in nbdkit when profiling nbdcopy?
There's no good reason. It's because the ./nbdkit wrapper is designed
for two purposes: To run nbdkit without installing from the local
directory, and as a way to run nbdkit during tests. In tests we want
to add malloc checking because it's an easy way to discover some
memory problems, but it's not useful for the other reason. I need to
work out some way to turn on malloc checking only when running the
tests.
> * The test harness is nbdkit-sparse-random-plugin, documented
here:
> https://libguestfs.org/nbdkit-sparse-random-plugin.1.html
Does it create a similar pattern to real world images, or more like
the worst case?
In my tests using nbdkit memory and pattern plugins was way more
stable compared with real images via qemu-nbd/nbdkit, but real image
give more real results :-)
It's meant to make virtual images which consist of runs of blocks and
holes, and therefore look at least somewhat like disk images:
https://gitlab.com/nbdkit/nbdkit/-/blob/65d0d80da15ed01ed1f9d82fef3165969...
The problem with using real disk images is they're huge, the test
tends to be dominated by other effects like disk I/O and memory, and
the results are not easily reproducible by other people. (With
sparse-random, setting the seed and other parameters should produce
bit-for-bit identical images). The problem with the memory and
pattern plugins is that the sparseness is unrealistic.
Maybe we can extract the extents from a real image, and add a plugin
accepting json extents and inventing data for the data extents?
Something like this would also work.
> * I'm using DWARF debugging info to generate call stacks,
which is
> more reliable than the default (frame pointers).
When I tried to use perf, I did not get proper call stacks, maybe this
was the reason.
Yes, perf is infuriatingly difficult to get reliable stacks. The
latest bug I'm wrestling with is:
https://bugzilla.redhat.com/show_bug.cgi?id=1963865
> * The -a option means I'm measuring events on the whole
machine. You
> can read the perf manual to find out how to measure only a single
> process (eg. just nbdkit or just nbdcopy). But actually measuring
> the whole machine gives a truer picture, I believe.
Why profile the whole machine? I would profile only nbdcopy or nbdkit,
depending on what we are trying to focus on.
Actually I think there are use cases for both. However if I'd only
profiled nbdcopy then I wouldn't have noticed that the sparse-plugin
had a horrible hotspot in one function! (now fixed)
Looking in the attached flame graph, if we focus on the nbdcopy
worker_thread, and sort by time:
poll_both_ends: 14.53% (58%)
malloc: 5.55% (22%)
nbd_ops_async_read: 4.34% (17%)
nbd_ops_get_extents: 0.52% (2%)
If we focus into into poll_both_ends:
send 10.17% (69%)
free 4.53% (31%)
So we have a lot of opportunities to optimize by allocating all
buffers up front as done in examples/libev-copy. But I'm not sure we
will would see the same picture when using smaller buffers
(--request-size 1m).
Also about 1/3rd of all the time is spent copying data to and from
userspace.
And even though as you point out, nbdcopy does too much malloc, it's
surprising that the overhead of malloc is so large.
So for context Abhay is investigating io_uring. I think this allows
us to register buffers with the kernel and have the kernel copy data
directly into and out of those buffers. If it's possible, this should
solve both problems.
One thing I'm not clear about is how exactly Unix domain sockets work.
Do they use packets and framing (and therefore skbs)? I kind of
assume not since it's just two processes on the same host producing
and consuming data. If skbs are involved in the network layer then
presumably fewer opportunities to do zero copy.
(Note that in the real world of virt-v2v and hypervisor migration we
expect to use a mix of Unix domain sockets and TCP, depending on the
exact source and destination of the copy).
nbd_ops_async_read is surprising - this is an async operation that
should
consume no time. Why does it take 17% of the time?
Yeah there are a few mysteries like this. More things for Abhay to
investigate ...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
5:25 a.m.
On Wed, May 26, 2021 at 10:32:08AM +0100, Richard W.M. Jones wrote:
On Wed, May 26, 2021 at 11:40:11AM +0300, Nir Soffer wrote:
> On Tue, May 25, 2021 at 9:06 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
> > I ran perf as below. Although nbdcopy and nbdkit themselves do not
> > require root (and usually should _not_ be run as root), in this case
> > perf must be run as root, so everything has to be run as root.
> >
> > # perf record -a -g --call-graph=dwarf ./nbdkit -U - sparse-random size=1T
--run "MALLOC_CHECK_= ../libnbd/run nbdcopy \$uri \$uri"
>
> This uses 64 requests with a request size of 32m. In my tests using
> --requests 16 --request-size 1048576 is faster. Did you try to profile
> this?
Interesting! No I didn't. In fact I just assumed that larger request
sizes / number of parallel requests would be better.
This is the topology of the machine I ran the tests on:
https://rwmj.files.wordpress.com/2019/09/screenshot_2019-09-04_11-08-41.png
Even a single 32MB buffer isn't going to fit in any cache, so reducing
buffer size should be a win, and once they are within the size of the
L3 cache, reusing buffers should also be a win.
That's the theory anyway ... Using --request-size=1048576 changes the
flamegraph quite dramatically (see new attachment).
[What is the meaning of the swapper stack traces? They are coming
from idle cores?]
Test runs slightly faster:
$ hyperfine 'nbdkit -U - sparse-random size=1T --run "nbdcopy \$uri
\$uri"'
Benchmark #1: nbdkit -U - sparse-random size=1T --run "nbdcopy \$uri \$uri"
Time (mean ± σ): 47.407 s ± 0.953 s [User: 347.982 s, System: 276.220 s]
Range (min … max): 46.474 s … 49.373 s 10 runs
$ hyperfine 'nbdkit -U - sparse-random size=1T --run "nbdcopy
--request-size=1048576 \$uri \$uri"'
Benchmark #1: nbdkit -U - sparse-random size=1T --run "nbdcopy
--request-size=1048576 \$uri \$uri"
Time (mean ± σ): 43.796 s ± 0.799 s [User: 328.134 s, System: 252.775 s]
Range (min … max): 42.289 s … 44.917 s 10 runs
(Note the buffers are still not being reused.)
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
6:50 a.m.
On Wed, May 26, 2021 at 1:25 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
On Wed, May 26, 2021 at 10:32:08AM +0100, Richard W.M. Jones wrote:
> On Wed, May 26, 2021 at 11:40:11AM +0300, Nir Soffer wrote:
> > On Tue, May 25, 2021 at 9:06 PM Richard W.M. Jones <rjones(a)redhat.com>
wrote:
> > > I ran perf as below. Although nbdcopy and nbdkit themselves do not
> > > require root (and usually should _not_ be run as root), in this case
> > > perf must be run as root, so everything has to be run as root.
> > >
> > > # perf record -a -g --call-graph=dwarf ./nbdkit -U - sparse-random
size=1T --run "MALLOC_CHECK_= ../libnbd/run nbdcopy \$uri \$uri"
> >
> > This uses 64 requests with a request size of 32m. In my tests using
> > --requests 16 --request-size 1048576 is faster. Did you try to profile
> > this?
>
> Interesting! No I didn't. In fact I just assumed that larger request
> sizes / number of parallel requests would be better.
This is the topology of the machine I ran the tests on:
https://rwmj.files.wordpress.com/2019/09/screenshot_2019-09-04_11-08-41.png
Even a single 32MB buffer isn't going to fit in any cache, so reducing
buffer size should be a win, and once they are within the size of the
L3 cache, reusing buffers should also be a win.
That's the theory anyway ... Using --request-size=1048576 changes the
flamegraph quite dramatically (see new attachment).
Interestingly, now malloc is about 35% of the time (6.6/18.4) of the
worker thread.
[What is the meaning of the swapper stack traces? They are coming
from idle cores?]
Test runs slightly faster:
$ hyperfine 'nbdkit -U - sparse-random size=1T --run "nbdcopy \$uri
\$uri"'
Benchmark #1: nbdkit -U - sparse-random size=1T --run "nbdcopy \$uri \$uri"
Time (mean ± σ): 47.407 s ± 0.953 s [User: 347.982 s, System: 276.220 s]
Range (min … max): 46.474 s … 49.373 s 10 runs
$ hyperfine 'nbdkit -U - sparse-random size=1T --run "nbdcopy
--request-size=1048576 \$uri \$uri"'
Benchmark #1: nbdkit -U - sparse-random size=1T --run "nbdcopy
--request-size=1048576 \$uri \$uri"
Time (mean ± σ): 43.796 s ± 0.799 s [User: 328.134 s, System: 252.775 s]
Range (min … max): 42.289 s … 44.917 s 10 runs
Adding --requests 16 is faster with real server, copying real images
and shared storage.
These flamegraphs are awesome!
Here are results from tests I did a few month ago in the RHV scale lab.
## Server
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
cores: 40
memory: 503g
## Source
Dell Express Flash PM1725b 3.2TB SFF
According to Dell site, this is:
http://image-us.samsung.com/SamsungUS/PIM/Samsung_1725b_Product.pdf
# qemu-img info /scratch/nsoffer-v2v.qcow2
image: /scratch/nsoffer-v2v.qcow2
file format: qcow2
virtual size: 100 GiB (107374182400 bytes)
disk size: 66.5 GiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
Exported with qemu-nbd:
qemu-nbd --persistent --shared=8 --format=qcow2 --cache=none
--aio=native --read-only /scratch/nsoffer-v2v.qcow2 --socket
/tmp/src.sock
(Using configuration used by oVirt when exporting disks for backup)
## Destination
NetApp LUN connected via FC via 4 paths:
# multipath -ll
3600a098038304437415d4b6a59682f76 dm-4 NETAPP,LUN C-Mode
size=1.0T features='3 queue_if_no_path pg_init_retries 50'
hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 8:0:1:0 sdf 8:80 active ready running
| `- 8:0:0:0 sdd 8:48 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 1:0:0:0 sde 8:64 active ready running
`- 1:0:1:0 sdg 8:96 active ready running
Disk is a logical volume on this lun:
# qemu-img info -U
/dev/f7b5c299-df2a-42bc-85d7-b60027f14e00/8825cff6-a9ef-4f8a-b159-97d77e21cf03
image: /dev/f7b5c299-df2a-42bc-85d7-b60027f14e00/8825cff6-a9ef-4f8a-b159-97d77e21cf03
file format: qcow2
virtual size: 100 GiB (107374182400 bytes)
disk size: 0 B
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
Exported with qemu-nbd:
qemu-nbd --persistent --shared=8 --format=qcow2 --cache=none
--aio=native /root/nsoffer/target-disk --socket /tmp/dst.sock
## Compare qemu-img convert, nbdcopy and libev-copy with similar
sparse settings.
Basically all give very similar results.
# hyperfine "./copy-libev $SRC $DST" "qemu-img convert -n -W -m 16 -S
1048576 $SRC $DST" "../copy/nbdcopy --sparse=1048576
--request-size=1048576 --flush --requests=16 --connections=1 $SRC
$DST"
Benchmark #1: ./copy-libev nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 103.514 s ± 0.836 s [User: 7.153 s, System: 19.422 s]
Range (min … max): 102.265 s … 104.824 s 10 runs
Benchmark #2: qemu-img convert -n -W -m 16 -S 1048576
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 103.104 s ± 0.899 s [User: 2.897 s, System: 25.204 s]
Range (min … max): 101.958 s … 104.499 s 10 runs
Benchmark #3: ../copy/nbdcopy --sparse=1048576 --request-size=1048576
--flush --requests=16 --connections=1
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 104.085 s ± 0.977 s [User: 7.188 s, System: 19.965 s]
Range (min … max): 102.314 s … 105.153 s 10 runs
Summary
'qemu-img convert -n -W -m 16 -S 1048576
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
ran
1.00 ± 0.01 times faster than './copy-libev
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
1.01 ± 0.01 times faster than '../copy/nbdcopy --sparse=1048576
--request-size=1048576 --flush --requests=16 --connections=1
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
## Compare nbdcopy request size with 16 requests and one connection
# hyperfine "./copy-libev nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock"
Benchmark #1: ./copy-libev nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 104.195 s ± 1.911 s [User: 8.652 s, System: 18.887 s]
Range (min … max): 102.474 s … 108.660 s 10 runs
# hyperfine -L r 524288,1048576,2097152 --export-json
nbdcopy-nbd-to-nbd-request-size.json "./nbdcopy --requests=16
--request-si
ze={r} nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock"
Benchmark #1: ./nbdcopy --requests=16 --request-size=524288
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 108.251 s ± 0.942 s [User: 5.538 s, System:
21.327 s]
Range (min … max): 107.098 s … 110.019 s 10 runs
Benchmark #2: ./nbdcopy --requests=16 --request-size=1048576
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 105.973 s ± 0.732 s [User: 7.901 s, System:
22.064 s]
Range (min … max): 104.915 s … 107.003 s 10 runs
Benchmark #3: ./nbdcopy --requests=16 --request-size=2097152
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 109.151 s ± 1.355 s [User: 9.898 s, System: 26.591 s]
Range (min … max): 107.168 s … 111.176 s 10 runs
Summary
'./nbdcopy --requests=16 --request-size=1048576
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
ran
1.02 ± 0.01 times faster than './nbdcopy --requests=16
--request-size=524288 nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock'
1.03 ± 0.01 times faster than './nbdcopy --requests=16
--request-size=2097152 nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock'
## Compare number of requests with multiple connections
To enable multiple connections to the destination, I hacked nbdcopy to
ignore the the
destination can_multicon always use multiple connections. This is how we use
qemu-nbd with imageio in RHV.
This shows 10% better performance, best with 4 requests per connection, but
the difference between 4,8, and 16 is not significant.
# hyperfine -r3 -L r 1,2,4,8,16 "./nbdcopy --flush
--request-size=1048576 --requests={r} --connections=4
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock"
Benchmark #1: ./nbdcopy --flush --request-size=1048576 --requests=1
--connections=4 nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 117.876 s ± 1.612 s [User: 6.968 s, System: 23.676 s]
Range (min … max): 116.163 s … 119.363 s 3 runs
Benchmark #2: ./nbdcopy --flush --request-size=1048576 --requests=2
--connections=4 nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 96.447 s ± 0.319 s [User: 8.216 s, System: 23.213 s]
Range (min … max): 96.192 s … 96.805 s 3 runs
Benchmark #3: ./nbdcopy --flush --request-size=1048576 --requests=4
--connections=4 nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 91.356 s ± 0.339 s [User: 10.269 s, System: 23.029 s]
Range (min … max): 91.013 s … 91.691 s 3 runs
Benchmark #4: ./nbdcopy --flush --request-size=1048576 --requests=8
--connections=4 nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 91.387 s ± 0.965 s [User: 12.699 s, System: 26.156 s]
Range (min … max): 90.786 s … 92.500 s 3 runs
Warning: Statistical outliers were detected. Consider re-running
this benchmark on a quiet PC without any interferences from other
programs. It might help to use the '--warmup' or '--prepare' options.
Benchmark #5: ./nbdcopy --flush --request-size=1048576 --requests=16
--connections=4 nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 91.637 s ± 0.861 s [User: 13.816 s, System: 31.043 s]
Range (min … max): 91.077 s … 92.629 s 3 runs
Summary
'./nbdcopy --flush --request-size=1048576 --requests=4
--connections=4 nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock' ran
1.00 ± 0.01 times faster than './nbdcopy --flush
--request-size=1048576 --requests=8 --connections=4
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
1.00 ± 0.01 times faster than './nbdcopy --flush
--request-size=1048576 --requests=16 --connections=4
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
1.06 ± 0.01 times faster than './nbdcopy --flush
--request-size=1048576 --requests=2 --connections=4
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
1.29 ± 0.02 times faster than './nbdcopy --flush
--request-size=1048576 --requests=1 --connections=4
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock'
Nir
8:03 a.m.
On Wed, May 26, 2021 at 02:50:32PM +0300, Nir Soffer wrote:
Basically all give very similar results.
# hyperfine "./copy-libev $SRC $DST" "qemu-img convert -n -W -m 16 -S
1048576 $SRC $DST" "../copy/nbdcopy --sparse=1048576
--request-size=1048576 --flush --requests=16 --connections=1 $SRC
$DST"
Benchmark #1: ./copy-libev nbd+unix:///?socket=/tmp/src.sock
nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 103.514 s ± 0.836 s [User: 7.153 s, System: 19.422 s]
Range (min … max): 102.265 s … 104.824 s 10 runs
Benchmark #2: qemu-img convert -n -W -m 16 -S 1048576
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 103.104 s ± 0.899 s [User: 2.897 s, System: 25.204 s]
Range (min … max): 101.958 s … 104.499 s 10 runs
Benchmark #3: ../copy/nbdcopy --sparse=1048576 --request-size=1048576
--flush --requests=16 --connections=1
nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
Time (mean ± σ): 104.085 s ± 0.977 s [User: 7.188 s, System: 19.965 s]
Range (min … max): 102.314 s … 105.153 s 10 runs
In my testing, nbdcopy is a clear 4x faster than qemu-img convert, with
4 also happening to be the default number of connections/threads.
Why use nbdcopy --connections=1? That completely disables threads in
nbdcopy. Also I'm not sure if --flush is fair (it depends on what
qemu-img does, which I don't know).
The other interesting things are the qemu-img convert flags you're using:
-m 16 number of coroutines, default is 8
-W out of order writes, but the manual says "This is only recommended
for preallocated devices like host devices or other raw block
devices" which is a very unclear recommendation to me.
What's special about host devices versus (eg) files or
qcow2 files which means -W wouldn't always be recommended?
Anyway I tried various settings to see if I could improve the
performance of qemu-img convert vs nbdcopy using the sparse-random
test harness. The results seem to confirm what has been said in this
thread so far.
libnbd-1.7.11-1.fc35.x86_64
nbdkit-1.25.8-2.fc35.x86_64
qemu-img-6.0.0-1.fc35.x86_64
$ hyperfine 'nbdkit -U - sparse-random size=100G --run "qemu-img convert \$uri
\$uri"' 'nbdkit -U - sparse-random size=100G --run "qemu-img convert -m
16 -W \$uri \$uri"' 'nbdkit -U - sparse-random size=100G --run "nbdcopy
\$uri \$uri"' 'nbdkit -U - sparse-random size=100G --run "nbdcopy
--request-size=1048576 --requests=16 \$uri \$uri"'
Benchmark #1: nbdkit -U - sparse-random size=100G --run "qemu-img convert \$uri
\$uri"
Time (mean ± σ): 17.245 s ± 1.004 s [User: 28.611 s, System: 7.219 s]
Range (min … max): 15.711 s … 18.930 s 10 runs
Benchmark #2: nbdkit -U - sparse-random size=100G --run "qemu-img convert -m 16 -W
\$uri \$uri"
Time (mean ± σ): 8.618 s ± 0.266 s [User: 33.091 s, System: 7.331 s]
Range (min … max): 8.130 s … 8.943 s 10 runs
Benchmark #3: nbdkit -U - sparse-random size=100G --run "nbdcopy \$uri \$uri"
Time (mean ± σ): 5.227 s ± 0.153 s [User: 34.299 s, System: 30.136 s]
Range (min … max): 5.049 s … 5.439 s 10 runs
Benchmark #4: nbdkit -U - sparse-random size=100G --run "nbdcopy
--request-size=1048576 --requests=16 \$uri \$uri"
Time (mean ± σ): 4.198 s ± 0.197 s [User: 32.105 s, System: 24.562 s]
Range (min … max): 3.868 s … 4.474 s 10 runs
Summary
'nbdkit -U - sparse-random size=100G --run "nbdcopy --request-size=1048576
--requests=16 \$uri \$uri"' ran
1.25 ± 0.07 times faster than 'nbdkit -U - sparse-random size=100G --run
"nbdcopy \$uri \$uri"'
2.05 ± 0.12 times faster than 'nbdkit -U - sparse-random size=100G --run
"qemu-img convert -m 16 -W \$uri \$uri"'
4.11 ± 0.31 times faster than 'nbdkit -U - sparse-random size=100G --run
"qemu-img convert \$uri \$uri"'
## Compare nbdcopy request size with 16 requests and one connection
This is testing 4 connections I think? Or is the destination not
advertising multi-conn?
## Compare number of requests with multiple connections
To enable multiple connections to the destination, I hacked nbdcopy
to ignore the the destination can_multicon always use multiple
connections. This is how we use qemu-nbd with imageio in RHV.
So qemu-nbd doesn't advertise multi-conn? I'd prefer if we fixed qemu-nbd.
Anyway, interesting stuff, thanks.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
8:49 a.m.
On Wed, May 26, 2021 at 4:03 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
On Wed, May 26, 2021 at 02:50:32PM +0300, Nir Soffer wrote:
> Basically all give very similar results.
>
> # hyperfine "./copy-libev $SRC $DST" "qemu-img convert -n -W -m 16
-S
> 1048576 $SRC $DST" "../copy/nbdcopy --sparse=1048576
> --request-size=1048576 --flush --requests=16 --connections=1 $SRC
> $DST"
> Benchmark #1: ./copy-libev nbd+unix:///?socket=/tmp/src.sock
> nbd+unix:///?socket=/tmp/dst.sock
> Time (mean ± σ): 103.514 s ± 0.836 s [User: 7.153 s, System: 19.422 s]
> Range (min … max): 102.265 s … 104.824 s 10 runs
>
> Benchmark #2: qemu-img convert -n -W -m 16 -S 1048576
> nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
> Time (mean ± σ): 103.104 s ± 0.899 s [User: 2.897 s, System: 25.204 s]
> Range (min … max): 101.958 s … 104.499 s 10 runs
>
> Benchmark #3: ../copy/nbdcopy --sparse=1048576 --request-size=1048576
> --flush --requests=16 --connections=1
> nbd+unix:///?socket=/tmp/src.sock nbd+unix:///?socket=/tmp/dst.sock
> Time (mean ± σ): 104.085 s ± 0.977 s [User: 7.188 s, System: 19.965 s]
> Range (min … max): 102.314 s … 105.153 s 10 runs
In my testing, nbdcopy is a clear 4x faster than qemu-img convert, with
4 also happening to be the default number of connections/threads.
Why use nbdcopy --connections=1? That completely disables threads in
nbdcopy.
Because qemu-nbd does not report multicon when writing, so practically
you get one nbd handle for writing.
Also I'm not sure if --flush is fair (it depends on what
qemu-img does, which I don't know).
qemu is flushing at the end of the operation. Not flushing is cheating :-)
The other interesting things are the qemu-img convert flags
you're using:
-m 16 number of coroutines, default is 8
We use 8 in RHV since the difference is very small, and when running
concurrent copies it does not matter. Since we use up to 64 concurrent
requests in nbdcopy, it is useful to compare similar setup in qemu.
-W out of order writes, but the manual says "This is only
recommended
for preallocated devices like host devices or other raw block
devices" which is a very unclear recommendation to me.
What's special about host devices versus (eg) files or
qcow2 files which means -W wouldn't always be recommended?
This is how RHV use qemu-img convert when copying to raw preallocated
volumes. Using -W can be up to 6x times faster. We use the same for imageio
for any type of disk. This is the reason I tested this way.
-W is equivalent to the nbdocpy multithreaded copy using a single connection.
qemu-img does N concurrent reads. If you don't specify -W, it writes the data
in the right order (based on offset). If a read did not finish, the copy loops
waits until the read complets before writing. This ensure exactly one concurrent
write, and it is much slower.
Anyway I tried various settings to see if I could improve the
performance of qemu-img convert vs nbdcopy using the sparse-random
test harness. The results seem to confirm what has been said in this
thread so far.
libnbd-1.7.11-1.fc35.x86_64
nbdkit-1.25.8-2.fc35.x86_64
qemu-img-6.0.0-1.fc35.x86_64
$ hyperfine 'nbdkit -U - sparse-random size=100G --run "qemu-img convert \$uri
\$uri"' 'nbdkit -U - sparse-random size=100G --run "qemu-img convert -m
16 -W \$uri \$uri"' 'nbdkit -U - sparse-random size=100G --run "nbdcopy
\$uri \$uri"' 'nbdkit -U - sparse-random size=100G --run "nbdcopy
--request-size=1048576 --requests=16 \$uri \$uri"'
Benchmark #1: nbdkit -U - sparse-random size=100G --run "qemu-img convert \$uri
\$uri"
Time (mean ± σ): 17.245 s ± 1.004 s [User: 28.611 s, System: 7.219 s]
Range (min … max): 15.711 s … 18.930 s 10 runs
Benchmark #2: nbdkit -U - sparse-random size=100G --run "qemu-img convert -m 16 -W
\$uri \$uri"
Time (mean ± σ): 8.618 s ± 0.266 s [User: 33.091 s, System: 7.331 s]
Range (min … max): 8.130 s … 8.943 s 10 runs
Benchmark #3: nbdkit -U - sparse-random size=100G --run "nbdcopy \$uri \$uri"
Time (mean ± σ): 5.227 s ± 0.153 s [User: 34.299 s, System: 30.136 s]
Range (min … max): 5.049 s … 5.439 s 10 runs
Benchmark #4: nbdkit -U - sparse-random size=100G --run "nbdcopy
--request-size=1048576 --requests=16 \$uri \$uri"
Time (mean ± σ): 4.198 s ± 0.197 s [User: 32.105 s, System: 24.562 s]
Range (min … max): 3.868 s … 4.474 s 10 runs
Summary
'nbdkit -U - sparse-random size=100G --run "nbdcopy --request-size=1048576
--requests=16 \$uri \$uri"' ran
1.25 ± 0.07 times faster than 'nbdkit -U - sparse-random size=100G --run
"nbdcopy \$uri \$uri"'
2.05 ± 0.12 times faster than 'nbdkit -U - sparse-random size=100G --run
"qemu-img convert -m 16 -W \$uri \$uri"'
4.11 ± 0.31 times faster than 'nbdkit -U - sparse-random size=100G --run
"qemu-img convert \$uri \$uri"'
This shows that nbdcopy works better when the latency is (practically)
zero, copying
data from memory to memory. This is useful for minimizing overhead in nbdcopy,
but when copying real images with real storage with direct I/O the
time to write the
data to storage hides everything else.
Would it be useful to add latency in the sparse-random plugin, so it behaves
more like real storage? (or maybe it is already possible with a filter?)
> ## Compare nbdcopy request size with 16 requests and one
connection
This is testing 4 connections I think? Or is the destination not
advertising multi-conn?
> ## Compare number of requests with multiple connections
>
> To enable multiple connections to the destination, I hacked nbdcopy
> to ignore the the destination can_multicon always use multiple
> connections. This is how we use qemu-nbd with imageio in RHV.
So qemu-nbd doesn't advertise multi-conn? I'd prefer if we fixed qemu-nbd.
Right, but according to Eric, advertising multi-conn is not safe with current
code.
But he confirmed that using multiple writers writing to distinct
ranges is safe, so
nbdcopy can use this now for extra 10% performance in real copies.
Nir
9:15 a.m.
On Wed, May 26, 2021 at 04:49:50PM +0300, Nir Soffer wrote:
On Wed, May 26, 2021 at 4:03 PM Richard W.M. Jones
<rjones(a)redhat.com> wrote:
> In my testing, nbdcopy is a clear 4x faster than qemu-img convert, with
> 4 also happening to be the default number of connections/threads.
> Why use nbdcopy --connections=1? That completely disables threads in
> nbdcopy.
Because qemu-nbd does not report multicon when writing, so practically
you get one nbd handle for writing.
Let's see if we can fix that. Crippling nbdcopy because of a missing
feature in qemu-nbd isn't right. I wonder what Eric's reasoning for
multi-conn not being safe is.
> Also I'm not sure if --flush is fair (it depends on what
> qemu-img does, which I don't know).
qemu is flushing at the end of the operation. Not flushing is cheating :-)
That's fair enough. I will add that flag to my future tests.
I also pushed these commits to disable malloc checking outside tests:
https://gitlab.com/nbdkit/libnbd/-/commit/88e72dcb1631b315957f5f98e3cdfcd...
https://gitlab.com/nbdkit/nbdkit/-/commit/6039780f3bb0617650fa1fa4c1399b0...
> The other interesting things are the qemu-img convert flags
you're using:
>
> -m 16 number of coroutines, default is 8
We use 8 in RHV since the difference is very small, and when running
concurrent copies it does not matter. Since we use up to 64 concurrent
requests in nbdcopy, it is useful to compare similar setup in qemu.
I'm not really clear on the relationship (in qemu-img) between number
of coroutines, number of pthreads and number of requests in flight.
At this rate I'm going to have to look at the source :-)
> -W out of order writes, but the manual says "This is
only recommended
> for preallocated devices like host devices or other raw block
> devices" which is a very unclear recommendation to me.
> What's special about host devices versus (eg) files or
> qcow2 files which means -W wouldn't always be recommended?
This is how RHV use qemu-img convert when copying to raw preallocated
volumes. Using -W can be up to 6x times faster. We use the same for imageio
for any type of disk. This is the reason I tested this way.
-W is equivalent to the nbdocpy multithreaded copy using a single connection.
qemu-img does N concurrent reads. If you don't specify -W, it writes
the data in the right order (based on offset). If a read did not
finish, the copy loops waits until the read complets before
writing. This ensure exactly one concurrent write, and it is much
slower.
Thanks - interesting. Still not sure why you wouldn't want to use
this flag all the time.
See also:
https://lists.nongnu.org/archive/html/qemu-discuss/2021-05/msg00070.html
...
This shows that nbdcopy works better when the latency is
(practically) zero, copying data from memory to memory. This is
useful for minimizing overhead in nbdcopy, but when copying real
images with real storage with direct I/O the time to write the data
to storage hides everything else.
Would it be useful to add latency in the sparse-random plugin, so it
behaves more like real storage? (or maybe it is already possible
with a filter?)
We could use one of these filters:
https://libguestfs.org/nbdkit-delay-filter.1.html
https://libguestfs.org/nbdkit-rate-filter.1.html
Something like "--filter=delay wdelay=1ms" might be more realistic.
To simulate NVMe we might need to be able to specify microseconds there.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
10:15 a.m.
On Wed, May 26, 2021 at 5:15 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
...
> > -W out of order writes, but the manual says "This
is only recommended
> > for preallocated devices like host devices or other raw block
> > devices" which is a very unclear recommendation to me.
> > What's special about host devices versus (eg) files or
> > qcow2 files which means -W wouldn't always be recommended?
>
> This is how RHV use qemu-img convert when copying to raw preallocated
> volumes. Using -W can be up to 6x times faster. We use the same for imageio
> for any type of disk. This is the reason I tested this way.
>
> -W is equivalent to the nbdocpy multithreaded copy using a single connection.
>
> qemu-img does N concurrent reads. If you don't specify -W, it writes
> the data in the right order (based on offset). If a read did not
> finish, the copy loops waits until the read complets before
> writing. This ensure exactly one concurrent write, and it is much
> slower.
Thanks - interesting. Still not sure why you wouldn't want to use
this flag all the time.
We started to use -W for this bug:
https://bugzilla.redhat.com/1511891#c57
The comment shows that this can be up to 6 times faster. When copying to NFS
the improvement was smaller so we felt it is safer to avoid this optimization.
But since then we use unordered writes in imageio and in virt-v2v rhv
upload (master)
so I think we can simplify and always use unordered writes.
...
4:36 p.m.
On Wed, May 26, 2021 at 03:15:13PM +0100, Richard W.M. Jones wrote:
On Wed, May 26, 2021 at 04:49:50PM +0300, Nir Soffer wrote:
> On Wed, May 26, 2021 at 4:03 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
> > In my testing, nbdcopy is a clear 4x faster than qemu-img convert, with
> > 4 also happening to be the default number of connections/threads.
> > Why use nbdcopy --connections=1? That completely disables threads in
> > nbdcopy.
>
> Because qemu-nbd does not report multicon when writing, so practically
> you get one nbd handle for writing.
Let's see if we can fix that. Crippling nbdcopy because of a missing
feature in qemu-nbd isn't right. I wonder what Eric's reasoning for
multi-conn not being safe is.
multi-conn implies that connection A writes, connection B flushes, and
connection C is then guaranteed to read what connection A wrote.
Furthermore, if client A and B plan on doing overlapping writes, the
presence of multi-conn means that whoever flushes last is guaranteed
to have that last write stick. Without multiconn, even if A writes, B
writes, B flushes, then A flushes, you can end up with A's data
(rather than B's) as the final contents on disk, because the separate
connections are allowed to have separate caching regions where the
order of flushes determines which cache (with potentially stale data)
gets flushed when. And note that the effect of overlapping writes may
happen even when your client requests are not overlapping: if client A
and B both write distinct 512 byte regions within a larger 4k page,
the server performing RMW caching of that page will behave as though
there are overlapping writes.
During nbdcopy or qemu-img convert, we aren't reading what we just
wrote and can easily arrange to avoid overlapping writes, so we don't
care about the bulk of the semantics of multi-conn (other than it is a
nice hint of a server that accepts multiple clients). So at the end
of the day, it boils down to:
If the server advertised multi-conn: connect multiple clients, then
when all of them are done writing, only ONE client has to flush, and
the flush will be valid for what all of the clients wrote.
If the server did not advertise multi-conn, but still allows multiple
clients: connect those clients, write to distinct areas (avoid
overlapping writes, and hopefully your writes are sized large enough
that you are also avoiding overlapping cache granularities); then when
all clients are finished writing, ALL of them must call flush
(ideally, only the first flush takes any real time, and the server can
optimize the later flushes as having nothing further to flush - but we
can't guarantee that).
New enough nbdkit also has the multi-conn filter where you can play
around with different multi-conn policies :)
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
5:11 p.m.
On Thu, May 27, 2021 at 12:36 AM Eric Blake <eblake(a)redhat.com> wrote:
On Wed, May 26, 2021 at 03:15:13PM +0100, Richard W.M. Jones wrote:
> On Wed, May 26, 2021 at 04:49:50PM +0300, Nir Soffer wrote:
> > On Wed, May 26, 2021 at 4:03 PM Richard W.M. Jones <rjones(a)redhat.com>
wrote:
> > > In my testing, nbdcopy is a clear 4x faster than qemu-img convert, with
> > > 4 also happening to be the default number of connections/threads.
> > > Why use nbdcopy --connections=1? That completely disables threads in
> > > nbdcopy.
> >
> > Because qemu-nbd does not report multicon when writing, so practically
> > you get one nbd handle for writing.
>
> Let's see if we can fix that. Crippling nbdcopy because of a missing
> feature in qemu-nbd isn't right. I wonder what Eric's reasoning for
> multi-conn not being safe is.
multi-conn implies that connection A writes, connection B flushes, and
connection C is then guaranteed to read what connection A wrote.
Furthermore, if client A and B plan on doing overlapping writes, the
presence of multi-conn means that whoever flushes last is guaranteed
to have that last write stick. Without multiconn, even if A writes, B
writes, B flushes, then A flushes, you can end up with A's data
(rather than B's) as the final contents on disk, because the separate
connections are allowed to have separate caching regions where the
order of flushes determines which cache (with potentially stale data)
gets flushed when. And note that the effect of overlapping writes may
happen even when your client requests are not overlapping: if client A
and B both write distinct 512 byte regions within a larger 4k page,
the server performing RMW caching of that page will behave as though
there are overlapping writes.
During nbdcopy or qemu-img convert, we aren't reading what we just
wrote and can easily arrange to avoid overlapping writes, so we don't
care about the bulk of the semantics of multi-conn (other than it is a
nice hint of a server that accepts multiple clients). So at the end
of the day, it boils down to:
If the server advertised multi-conn: connect multiple clients, then
when all of them are done writing, only ONE client has to flush, and
the flush will be valid for what all of the clients wrote.
If the server did not advertise multi-conn, but still allows multiple
clients: connect those clients, write to distinct areas (avoid
overlapping writes, and hopefully your writes are sized large enough
that you are also avoiding overlapping cache granularities); then when
all clients are finished writing, ALL of them must call flush
(ideally, only the first flush takes any real time, and the server can
optimize the later flushes as having nothing further to flush - but we
can't guarantee that).
In nbdcopy case, all writers write 128mb segments, except the last
one which may have shorter segment, but this cannot overlap with
anything. So I think we cannot have overlapping writes due to partial
blocks.
Optimizing flush to do only one flush when server supports multi-conn
seems like very low priority to me, so we do this later.
Looks liks nbd_ops_flush is already doing the right thing, flushing all
open handles.
So the only change needed seems to be:
diff --git a/copy/main.c b/copy/main.c
index b9dbe1d..924fedb 100644
--- a/copy/main.c
+++ b/copy/main.c
@@ -321,10 +321,6 @@ main (int argc, char *argv[])
exit (EXIT_FAILURE);
}
- /* If multi-conn is not supported, force connections to 1. */
- if (! src->ops->can_multi_conn (src) || ! dst->ops->can_multi_conn (dst))
- connections = 1;
-
/* Calculate the number of threads from the number of connections. */
if (threads == 0) {
long t;
We always use the requested number of connections, ignoring can_multi_conn.
What do you think?
5:16 p.m.
On Thu, May 27, 2021 at 1:11 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Thu, May 27, 2021 at 12:36 AM Eric Blake <eblake(a)redhat.com> wrote:
>
> On Wed, May 26, 2021 at 03:15:13PM +0100, Richard W.M. Jones wrote:
> > On Wed, May 26, 2021 at 04:49:50PM +0300, Nir Soffer wrote:
> > > On Wed, May 26, 2021 at 4:03 PM Richard W.M. Jones
<rjones(a)redhat.com> wrote:
> > > > In my testing, nbdcopy is a clear 4x faster than qemu-img convert,
with
> > > > 4 also happening to be the default number of connections/threads.
> > > > Why use nbdcopy --connections=1? That completely disables threads
in
> > > > nbdcopy.
> > >
> > > Because qemu-nbd does not report multicon when writing, so practically
> > > you get one nbd handle for writing.
> >
> > Let's see if we can fix that. Crippling nbdcopy because of a missing
> > feature in qemu-nbd isn't right. I wonder what Eric's reasoning for
> > multi-conn not being safe is.
>
> multi-conn implies that connection A writes, connection B flushes, and
> connection C is then guaranteed to read what connection A wrote.
> Furthermore, if client A and B plan on doing overlapping writes, the
> presence of multi-conn means that whoever flushes last is guaranteed
> to have that last write stick. Without multiconn, even if A writes, B
> writes, B flushes, then A flushes, you can end up with A's data
> (rather than B's) as the final contents on disk, because the separate
> connections are allowed to have separate caching regions where the
> order of flushes determines which cache (with potentially stale data)
> gets flushed when. And note that the effect of overlapping writes may
> happen even when your client requests are not overlapping: if client A
> and B both write distinct 512 byte regions within a larger 4k page,
> the server performing RMW caching of that page will behave as though
> there are overlapping writes.
>
> During nbdcopy or qemu-img convert, we aren't reading what we just
> wrote and can easily arrange to avoid overlapping writes, so we don't
> care about the bulk of the semantics of multi-conn (other than it is a
> nice hint of a server that accepts multiple clients). So at the end
> of the day, it boils down to:
>
> If the server advertised multi-conn: connect multiple clients, then
> when all of them are done writing, only ONE client has to flush, and
> the flush will be valid for what all of the clients wrote.
>
> If the server did not advertise multi-conn, but still allows multiple
> clients: connect those clients, write to distinct areas (avoid
> overlapping writes, and hopefully your writes are sized large enough
> that you are also avoiding overlapping cache granularities); then when
> all clients are finished writing, ALL of them must call flush
> (ideally, only the first flush takes any real time, and the server can
> optimize the later flushes as having nothing further to flush - but we
> can't guarantee that).
In nbdcopy case, all writers write 128mb segments, except the last
one which may have shorter segment, but this cannot overlap with
anything. So I think we cannot have overlapping writes due to partial
blocks.
Optimizing flush to do only one flush when server supports multi-conn
seems like very low priority to me, so we do this later.
Looks liks nbd_ops_flush is already doing the right thing, flushing all
open handles.
So the only change needed seems to be:
diff --git a/copy/main.c b/copy/main.c
index b9dbe1d..924fedb 100644
--- a/copy/main.c
+++ b/copy/main.c
@@ -321,10 +321,6 @@ main (int argc, char *argv[])
exit (EXIT_FAILURE);
}
- /* If multi-conn is not supported, force connections to 1. */
- if (! src->ops->can_multi_conn (src) || ! dst->ops->can_multi_conn (dst))
- connections = 1;
-
/* Calculate the number of threads from the number of connections. */
if (threads == 0) {
long t;
Missing the next chunk:
@@ -384,10 +380,8 @@ main (int argc, char *argv[])
*/
if (connections > 1) {
assert (threads == connections);
- if (src->ops->can_multi_conn (src))
- src->ops->start_multi_conn (src);
- if (dst->ops->can_multi_conn (dst))
- dst->ops->start_multi_conn (dst);
+ src->ops->start_multi_conn (src);
+ dst->ops->start_multi_conn (dst);
}
/* If the source is NBD and we couldn't negotiate meta
We always use the requested number of connections, ignoring can_multi_conn.
What do you think?
1157
days inactive
1158
days old
12 comments
4 participants
participants (4)
-
Abhay Raj Singh -
Eric Blake -
Nir Soffer -
Richard W.M. Jones