On Wed, Jan 22, 2020 at 02:40:37PM +0000, Richard W.M. Jones wrote:
This is a proposal for an NBD to/from file copying tool (not actually
written). Obviously this would duplicate functionality which is
already available in qemu-img convert.
The reasons for writing this tool would be:
- to produce a tool which is very focused on the specific needs of
virt-v2v and similar migration scenarios
- to have a small tool with minimal dependencies
- fix some of the obvious problems with qemu-img convert like how it
handles devices which are already zeroed, and preallocation
- enable optimizations which make sense for NBD but which might not
be applicable to qemu (eg. multi-conn, freely writing out of order
from multiple threads).
Alternative is of course to not write a new tool and instead try to
fix all this stuff in qemu-img itself.
Anyway let me know your thoughts.
I like the proposal, there is only one thing I do not completely understand,
whether `-a sparse -S 0` means to just preserver the sparseness based on
whatever the server provides.
It would also be nice to have an option to specify a list of blocks to copy from
a file for example. Although it might be a niche feature and people should just
use libnbd at that point.
Rich.
nbdcp(1) LIBNBD nbdcp(1)
NAME
nbdcp - copy between NBD servers and local files
SYNOPSIS
nbdcp [-a|--target-allocation allocated|sparse]
[-m|--multi-conn <n>] [-M|--multi-conn-target <n>]
[-p|--progress-bar] [-S|--sparse-detect <n>]
[-T|--threads <n>] [-z|--target-is-zero]
'nbd://...'|DISK.IMG 'nbd://...'|DISK.IMG
DESCRIPTION
nbdcp is a utility that can copy quickly between NBD servers and local
raw format files (or block devices). It can copy:
from NBD server to file (or block device)
For example, this command copies from the NBD server listening on
port 10809 on "example.com" to a local file called disk.img:
nbdcp
nbd://example.com disk.img
from file (or block device) to NBD server
For example, this command copies from a local block device /dev/sda
to the NBD server listening on Unix domain socket /tmp/socket:
nbdcp /dev/sda 'nbd+unix:///?socket=/tmp/socket'
from NBD server to NBD server
For example this copies between two different exports on the same
NBD server:
nbdcp
nbd://example.com/export1 nbd://example.com/export2
This program cannot: copy from file to file (use cp(1) or dd(1)), copy
to or from formats other than raw (use qemu-img(1) convert), or access
servers other than NBD servers (also use qemu-img(1)).
NBD servers are specified by their URI, following the NBD URI standard
at
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md
Controlling sparseness or preallocation in the target
The options -a (--target-allocation), -S (--sparse-detect) and -z
(--target-is-zero) together control sparseness in the target file.
By default nbdcp tries to both preserve sparseness from the source and
will detect runs of allocated zeroes and turn them into sparseness. To
turn off detection of sparseness use "-S 0".
The -z option should be used if and only if you know that the target
block device is zeroed already. This allows an important optimization
where nbdcp can skip zeroing or trimming parts of the disk that are
already zero.
The -a option is used to control the desired final preallocation state
of the target. The default is "-a sparse" which makes the target as
sparse as possible. "-a allocated" makes the target fully allocated.
OPTIONS
--help
Display brief command line help and exit.
-a allocated
--target-allocation=allocated
Make the target fully allocated.
-a sparse
--target-allocation=sparse
Make the target as sparse as possible. This is the default. See
also "Controlling sparseness or preallocation in the target".
-m N
--multi-conn=N
Enable NBD multi-conn with up to "N" connections. Only some NBD
servers support this but it can greatly improve performance.
The default is to enable multi-conn if we detect that the server
supports it, with up to 4 connections.
-M N
--multi-conn-target=N
If you are copying between NBD servers, use -m to control the
multi-conn setting for the source server, and this option (-M) to
control the multi-conn setting for the target server.
-p
--progress-bar
Display a progress bar during copying.
-p machine:FD
--progress-bar=machine:FD
Write a machine-readable progress bar to file descriptor "FD".
This progress bar prints lines with the format "COPIED/TOTAL"
(where "COPIED" and "TOTAL" are 64 bit unsigned integers).
-S 0
--sparse-detect=0
Turn off sparseness detection.
-S N
--sparse-detect=N
Detect runs of zero bytes of at least size "N" bytes and turn them
into sparse blocks on the target (if "-a sparse" is used). This is
the default, with a 512 byte block size.
-T N
--threads N
Use at most "N" threads when copying. Usually more threads leads
to better performance, up to the limit of the number of cores on
your machine and the parallelism of the underlying disk or network.
The default is to use the number of online processors.
-z
--target-is-zero
Declare that the target block device contains only zero bytes (or
sparseness that reads back as zeroes). You must only use this
option if you are sure that this is true, since it means that nbdcp
will enable an optimization where it skips zeroing parts of the
disk that are zero on the source.
-V
--version
Display the package name and version and exit.
SEE ALSO
qemu-img(1), libnbd(3), nbdsh(1).
AUTHORS
Richard W.M. Jones
COPYRIGHT
Copyright (C) 2020 Red Hat Inc.
LICENSE
This library is free software; you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License as published
by the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This library is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
02110-1301 USA
libnbd-1.3.1 2020-01-22 nbdcp(1)