On Sat, May 29, 2021 at 09:33:58PM +0530, Abhay Raj Singh wrote: Hi Abhay, I think you'll fine the mailing list has a limit on the size of attachemnts it can accept so they probably won't appear in the mailing list archives. I looked at the SVGs anyhow and there is quite a bit of difference, which is possibly unexpected. It's useful to post the exact command line that you used, in case we're using different flags. Some observations: I think you're using swap-on-zram. (https://fedoraproject.org/wiki/Changes/SwapOnZRAM). The kswapd process is certainly spending a lot of time in zstd* functions. I'm not using zram. There's nothing necessarily wrong with this, it's just a difference in configuration. However ... Ideally the test should not be swapping at all, so it's unclear why pageout() is being called. How much RAM does the machine have? The kernel function asm_exc_page_fault appears a lot, most prominently for nbdcopy but also for nbdkit. This indicates that the x86 page fault trap happens a lot, which indicates that the processes are being swapped in and out frequently (https://elixir.bootlin.com/linux/latest/source/arch/x86/kernel/idt.c#L221) In my trace, about 40% of the time is seen in swapper. However I believe this indicates CPU cores which are idle (and my test machine has a lot of cores). swapper is hardly seen in your trace, which just means fewer cores are idle. It's not really a significant difference. My conclusion is the test machine probably doesn't have as much free memory, and there's a lot of swapping going on. While the test is running, try also running: "vmstat 1". If the "si" and "so" fields in the output are non-zero, then the machine is swapping. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

On Sun, May 30, 2021 at 12:13:28AM +0530, Abhay Raj Singh wrote: I think this comes back to something that Nir Soffer said last week about number of requests and request size: https://listman.redhat.com/archives/libguestfs/2021-May/msg00121.html I followed up with a discussion about L3 cache size vs default buffer size: https://listman.redhat.com/archives/libguestfs/2021-May/msg00122.html https://listman.redhat.com/archives/libguestfs/2021-May/msg00123.html In short the default settings for nbdcopy are wrong and more performance could be gained by having better settings. The question (for you!) is how we can choose better settings, and whether we should choose simply better defaults or have dynamic settings. My vision for nbdcopy is that it should work "as well as possible" without end users needing to change the settings on the command line. But there may be cases where users need to or want to tune particular settings. But I'm not sure how to achieve that. eg: Should we choose defaults based on the number of cores and amount of memory detected at run time? Or is that too complicated, and maybe we can choose more conservative defaults? Really I'm looking for your answers here. I do believe that qemu gets this wrong. There are a bunch of command line parameters that you have to set to get good performance, and they're not even documented. I would imagine that the large default buffer size x number of buffers would cause problems no matter the size of the virtual disk. [...] My guess is sampling rate would not affect things very much, but I didn't measure it so I could be wrong. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

In my opinion configuration resolution will be best if resolved in the following order of increasing priority/weightage. 1. Auto-detect depending on the system parameters. Auto-detection of best params is a must IMO as there's enough diversity in server hardware that one size won't fit all can be implemented without much effort(excluding benchmarking). 2. Configuration file: A yaml, toml, or any other simple format - Load a different config file with *nbdcopy --file=custom-config.yaml* - Maybe generate the default config file having values from the auto-detect phase. nbdcopy --generate-config ~/.config/nbdcopy/config.yaml (used to get values) ~/.config/nbdcopy/example.yaml (generated for reference from auto) 3. CLI params A quick way to override configuration file params ex: functionally same `nbdcopy --requests=4 --config=custom-config.yaml` `nbdcopy --config=custom-config.yaml --requests=4` The flow of initialization: Read config file ----override ---> CLI params --- missing params --> auto-detect fill or easier implementation: auto-detect fill --- override ---> config file --- override ---> CLI params The rationale for choosing configuration file method: - Easier to use avoiding long commands (less typing, scripts easier to read) - Easy switch between multiple configurations and override them. - Documentation and configuration in the same place So easier for someone else to understand why certain parameters were chosen.

On Sun, May 30, 2021 at 09:27:18PM +0530, Abhay Raj Singh wrote: This part I thoroughly agree with. But my argument is that "nbdcopy source dest" should do the right thing, and in normal circumstances we try to make it as efficient as reasonably possible. For scripts and programs that consume nbdcopy and that may need to add extra command line parameters, CLI construction/length is not an issue. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW