libnbd flamegraph Odd benchmark results
Rich, *nbdcopy3-original.svg* is the latest one you sent *nbdcopy-abhay.svg* is the one I was able to create using given commands. However, there was some noise when I compared my results with the original. Is it normal or should I try fixing it. Thanks! Regards, Abhay
On Sat, May 29, 2021 at 09:33:58PM +0530, Abhay Raj Singh wrote:
Rich, *nbdcopy3-original.svg* is the latest one you sent *nbdcopy-abhay.svg* is the one I was able to create using given commands.
However, there was some noise when I compared my results with the original. Is it normal or should I try fixing it. Thanks!
Hi Abhay, I think you'll fine the mailing list has a limit on the size of attachemnts it can accept so they probably won't appear in the mailing list archives. I looked at the SVGs anyhow and there is quite a bit of difference, which is possibly unexpected. It's useful to post the exact command line that you used, in case we're using different flags. Some observations: I think you're using swap-on-zram. (https://fedoraproject.org/wiki/Changes/SwapOnZRAM). The kswapd process is certainly spending a lot of time in zstd* functions. I'm not using zram. There's nothing necessarily wrong with this, it's just a difference in configuration. However ... Ideally the test should not be swapping at all, so it's unclear why pageout() is being called. How much RAM does the machine have? The kernel function asm_exc_page_fault appears a lot, most prominently for nbdcopy but also for nbdkit. This indicates that the x86 page fault trap happens a lot, which indicates that the processes are being swapped in and out frequently (https://elixir.bootlin.com/linux/latest/source/arch/x86/kernel/idt.c#L221) In my trace, about 40% of the time is seen in swapper. However I believe this indicates CPU cores which are idle (and my test machine has a lot of cores). swapper is hardly seen in your trace, which just means fewer cores are idle. It's not really a significant difference. My conclusion is the test machine probably doesn't have as much free memory, and there's a lot of swapping going on. While the test is running, try also running: "vmstat 1". If the "si" and "so" fields in the output are non-zero, then the machine is swapping. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
That might be the reason I have 8GB(7.7) of ram I verify 7.4 available by running in tty only mode and it gets filled pretty quickly (vmstat) Strange thing is that this also happens at size=1G. Command I am running is sudo perf record -a -g --call-graph=dwarf -F 300 ./sparse-random-test.sh ``` libnbd/profiling/sparse-random-test.sh ../../nbdkit/nbdkit -U - sparse-random size=16G --run "MALLOC_CHECK_= ../run nbdcopy \$uri \$uri" ``` -F doesn't affect the result much, it decreases the sampling rate to avoid CPU/IO overload caused by perf. On a related note, what should I fill as server requirements in the OSU open source lab application asking for a test server? I was thinking that a dedicated server unit with 16-32GB, 1TB should be enough. Regards, Abhay
On Sun, May 30, 2021 at 12:13:28AM +0530, Abhay Raj Singh wrote:
That might be the reason
I have 8GB(7.7) of ram I verify 7.4 available by running in tty only mode and it gets filled pretty quickly (vmstat)
I think this comes back to something that Nir Soffer said last week about number of requests and request size: https://listman.redhat.com/archives/libguestfs/2021-May/msg00121.html I followed up with a discussion about L3 cache size vs default buffer size: https://listman.redhat.com/archives/libguestfs/2021-May/msg00122.html https://listman.redhat.com/archives/libguestfs/2021-May/msg00123.html In short the default settings for nbdcopy are wrong and more performance could be gained by having better settings. The question (for you!) is how we can choose better settings, and whether we should choose simply better defaults or have dynamic settings. My vision for nbdcopy is that it should work "as well as possible" without end users needing to change the settings on the command line. But there may be cases where users need to or want to tune particular settings. But I'm not sure how to achieve that. eg: Should we choose defaults based on the number of cores and amount of memory detected at run time? Or is that too complicated, and maybe we can choose more conservative defaults? Really I'm looking for your answers here. I do believe that qemu gets this wrong. There are a bunch of command line parameters that you have to set to get good performance, and they're not even documented.
Strange thing is that this also happens at size=1G.
I would imagine that the large default buffer size x number of buffers would cause problems no matter the size of the virtual disk. [...]
-F doesn't affect the result much, it decreases the sampling rate to avoid CPU/ IO overload caused by perf.
My guess is sampling rate would not affect things very much, but I didn't measure it so I could be wrong. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Sat, May 29, 2021 at 09:46:08PM +0100, Richard W.M. Jones wrote:
In short the default settings for nbdcopy are wrong and more performance could be gained by having better settings.
To be clearer, I mean that if we allocate (default) 32MB buffers and have (default) 64 requests in flight and have (default) 4 connections open, then in theory there could be 32 * 64 * 4 MB of memory being used at once. That is 8 GB -- although in reality it would probably not be that much. Also each single buffer is larger than my test machine's L3 cache. Then there's the problem that Nir also pointed out that we don't reuse these memory buffers, so we depend on whatever malloc() does, and may or may not be reusing the memory in a cache-efficient way. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
The question (for you!) is how we can choose better settings, and whether we should choose
In my opinion configuration resolution will be best if resolved in the following order of increasing priority/weightage. 1. Auto-detect depending on the system parameters. Auto-detection of best params is a must IMO as there's enough diversity in server hardware that one size won't fit all can be implemented without much effort(excluding benchmarking). 2. Configuration file: A yaml, toml, or any other simple format - Load a different config file with *nbdcopy --file=custom-config.yaml* - Maybe generate the default config file having values from the auto-detect phase. nbdcopy --generate-config ~/.config/nbdcopy/config.yaml (used to get values) ~/.config/nbdcopy/example.yaml (generated for reference from auto) 3. CLI params A quick way to override configuration file params ex: functionally same `nbdcopy --requests=4 --config=custom-config.yaml` `nbdcopy --config=custom-config.yaml --requests=4` The flow of initialization: Read config file ----override ---> CLI params --- missing params --> auto-detect fill or easier implementation: auto-detect fill --- override ---> config file --- override ---> CLI params The rationale for choosing configuration file method: - Easier to use avoiding long commands (less typing, scripts easier to read) - Easy switch between multiple configurations and override them. - Documentation and configuration in the same place So easier for someone else to understand why certain parameters were chosen.
On Sun, May 30, 2021 at 09:27:18PM +0530, Abhay Raj Singh wrote:
The question (for you!) is how we can choose better settings, and whether we should choose
In my opinion configuration resolution will be best if resolved in the following order of increasing priority/weightage. 1. Auto-detect depending on the system parameters. Auto-detection of best params is a must IMO as there's enough diversity in server hardware that one size won't fit all can be implemented without much effort(excluding benchmarking).
What are the inputs to this formula? I think the only ones we can portably get are: number of cores, total memory. Free memory possibly, although it's a slippery concept (is that "free" before or after the page cache is freed?) Other inputs that could make a difference, such as number of NUMA nodes, size of L2/L3 cache, core clock rate, ... are difficult to get reliably and portably. Still, if the formula is clear and understandable to users it could be fine.
2. Configuration file: A yaml, toml, or any other simple format - Load a different config file with *nbdcopy --file=custom-config.yaml* - Maybe generate the default config file having values from the auto-detect phase. nbdcopy --generate-config
~/.config/nbdcopy/config.yaml (used to get values) ~/.config/nbdcopy/example.yaml (generated for reference from auto)
I think this part is a horrible idea for several reasons: - Adds hidden state: As an end user you need to know the configuration file exists. And if end users have problems we need them to send us the configuration file, and my experience with virt-v2v tells me that follow-up questions about bugs are rarely answered. - Scripts will behave differently depending on where they are running. - Any script or program consuming nbdcopy must disable the config file processing. It will be completely unwanted for them to depend on the random contents of a config file that could greatly affect (or even break) the program. - Adds much complexity to the implementation of the tool. - Yaml etc are awful formats for configuration files in general. Rich.
3. CLI params A quick way to override configuration file params ex: functionally same `nbdcopy --requests=4 --config=custom-config.yaml` `nbdcopy --config=custom-config.yaml --requests=4`
The flow of initialization: Read config file ----override ---> CLI params --- missing params --> auto-detect fill or easier implementation: auto-detect fill --- override ---> config file --- override ---> CLI params
The rationale for choosing configuration file method: - Easier to use avoiding long commands (less typing, scripts easier to read) - Easy switch between multiple configurations and override them. - Documentation and configuration in the same place So easier for someone else to understand why certain parameters were chosen.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Sun, May 30, 2021 at 09:27:18PM +0530, Abhay Raj Singh wrote:
- Easier to use avoiding long commands (less typing, scripts easier to read)
This part I thoroughly agree with. But my argument is that "nbdcopy source dest" should do the right thing, and in normal circumstances we try to make it as efficient as reasonably possible. For scripts and programs that consume nbdcopy and that may need to add extra command line parameters, CLI construction/length is not an issue. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
participants (2)
-
Abhay Raj Singh -
Richard W.M. Jones