On Mon, Mar 23, 2015 at 04:34:21PM +0200, NoxDaFox wrote: > Greetings, > > I have the following typical scenario: given one or more qcow2 base images > I clone them with COW and start the VMs. > > At a certain point I'd like to inspect them in order to see their evolution > compared to the known base images. To do so I was thinking about taking a > disk snapshot of each VM and inspect its content through libguestfs (using > it's Python bindings). > > Obviously I need the base image in order for libguestfs to correctly guess > the OS, the FS structure etc.. Problem is that that point when I inspect > the disk I get the whole disk state including the base image content (30K+ > files and directories). > > This is not an issue but it's a very heavy operation considering that some > of the snapshots are few megabytes while the base images are several > gigabytes. > > Is there a way to programmatically instruct libguestfs to limit the > inspection to the sole snapshot? > Would it work as well with other disk format (vmdk linked clones for > example)? > > For better comprehension I'll show the sequence I'm doing (I might do it > wrong of course): > > virsh -c "qemu:///system" snapshot-create --disk-only <domain-ID> > > I get the snapshot location from its XML description and then: > > qemu-img convert -f qcow2 -O qcow2 base_image.qcow2 snapshot.qcow2 This makes a copy of the whole disk image. It's also not a consistent (point in time) copy.

> At that point I mount it through libguestfs and inspect its content. As long as you use the 'readonly=1' flag (which is really *essential*, else you'll get disk corruption), you can just point libguestfs at the base image: g = guestfs.GuestFS (python_return_dict=True) g.add_drive_opts ("base_image.qcow2", format="qcow2", readonly=1) That also doesn't get you a consistent snapshot, but it'll work most of the time, and give you a clear error in libguestfs when it doesn't (and won't corrupt your base disk or anything like that, provided you're using readonly=1). The effect of the readonly=1 flag is to create an external snapshot. It is roughly the equivalent of doing: qemu-img create -f qcow2 -b base_image snapshot.qcow2 < point libguestfs at snapshot.qcow2 > If you want lightweight, consistent, point-in-time snapshots (which it sounds like you do), qemu has recently been adding this capability. See the 'drive-backup' monitor command. I've not tried using that and I don't know if it is wired up through libvirt, but libguestfs should be able to consume it since it's just an NBD source.

On Mon, Mar 23, 2015 at 10:43:30PM +0000, Richard W.M. Jones wrote: [. . .] > > This makes a copy of the whole disk image. It's also not a consistent > > (point in time) copy. > > Oh I see that you're copying the _snapshot_ that you created with > libvirt; it's not a whole disk copy. There's still not any point in > doing this, and what I said below stands. > > > > At that point I mount it through libguestfs and inspect its content. > > > > As long as you use the 'readonly=1' flag (which is really *essential*, > > else you'll get disk corruption), you can just point libguestfs at the > > base image: > > > > g = guestfs.GuestFS (python_return_dict=True) > > g.add_drive_opts ("base_image.qcow2", format="qcow2", readonly=1) > > > > That also doesn't get you a consistent snapshot, but it'll work most > > of the time, and give you a clear error in libguestfs when it doesn't > > (and won't corrupt your base disk or anything like that, provided > > you're using readonly=1). > > > > The effect of the readonly=1 flag is to create an external snapshot. > > It is roughly the equivalent of doing: > > > > qemu-img create -f qcow2 -b base_image snapshot.qcow2 > > < point libguestfs at snapshot.qcow2 > > > > > If you want lightweight, consistent, point-in-time snapshots (which it > > sounds like you do), qemu has recently been adding this capability. > > See the 'drive-backup' monitor command. I've not tried using that A small QMP test for 'drive-backup': https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt

From what I gather, drive-backup would take a nbd:... target, meaning

[Added Stefan Hajnoczi, who wrote the 'drive-backup' QMP command and Eric Blake.] On Wed, Mar 25, 2015 at 07:41:23PM +0000, Richard W.M. Jones wrote: > On Wed, Mar 25, 2015 at 07:38:03PM +0100, Kashyap Chamarthy wrote: > > On Mon, Mar 23, 2015 at 10:43:30PM +0000, Richard W.M. Jones wrote: [. . .] > > > > If you want lightweight, consistent, point-in-time snapshots (which it > > > > sounds like you do), qemu has recently been adding this capability. Yeah, I have the two-part incremental backup series from John Snow's (who is the author of it) git repo. But, he said one more revision is about to hit the list, so I held off my testing until I can cleanly test that series on top of current QEMU git master. > > > > See the 'drive-backup' monitor command. I've not tried using that > > > > A small QMP test for 'drive-backup': > > > > https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt > > That's interesting, but even more interesting would be to see > drive-backup writing to NBD. > > From what I gather, drive-backup would take a nbd:... target, meaning > that it is the *client* (not the server). You're right -- the original cover letter[*] of the 'drive-backup' QEMU patch series that was merged says it: "The simplest use-case is to copy a point-in-time snapshot to a local file. More advanced users may wish to make the target an NBD URL. The NBD server listening on the other side can process the backup writes any way it wishes." > What's interesting is what you're supposed to connect to the server. > (Maybe I'm wrong about this however) Yeah, I too wonder what to connect on the server side when using NBD as target with 'drive-backup' (taking the simple example test I pointed above). So, something like? . . . { 'execute': 'drive-backup', 'arguments': { 'device': 'drive-virtio-disk1', 'sync': 'full', 'target': 'nbd://localhost', 'mode': 'absolute-paths', 'format': 'qcow2' } . . . Same question as yours, what is the NBD server going to run? My only NBD testing so far has been with w/ NBD over Unix socket or over TCP[**].

[*] http://lists.nongnu.org/archive/html/qemu-devel/2013-06/msg04229.html [**] NBD over TCP: (1) Run an NBD server locally: $ qemu-nbd -f qcow2 -p10809 \ /var/lib/libvirt/images/cirros-0.3.3-x86_64-disk.img -t (2) Create an overlay using the NBD server as backing file: $ qemu-img create -f qcow2 -F raw \ -o backing_file=nbd://localhost nbd-overlay1.qcow2 (3) Optionally, boot the overlay w/ a minimal CLI. $ qemu-system-x86_64 \ -nographic \ -nodefconfig \ -nodefaults \ -m 2048 \ -device virtio-scsi-pci,id=scsi \ -device virtio-serial-pci \ -serial stdio \ -drive file=./nbd-overlay1.qcow2,format=qcow2,if=virtio,cache=writeback -- /kashyap

On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi wrote: > On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy wrote: > > So, something like? > > > > . . . > > { 'execute': 'drive-backup', 'arguments': > > { 'device': 'drive-virtio-disk1', 'sync': 'full', 'target': > > 'nbd://localhost', 'mode': 'absolute-paths', 'format': 'qcow2' } > > . . . > > > > Same question as yours, what is the NBD server going to run? > > > > > > My only NBD testing so far has been with w/ NBD over Unix socket or > > over TCP[**]. > > For 'sync': 'full' mode qemu-nbd or nbd-server can be used as the > server. You probably don't want 'format': 'qcow2', just raw data over > NBD. That way the NBD server can implement whatever storage backend it > likes (raw, qcow2, something else). > > For incremental backup the NBD server would either be a qemu-nbd > instance with a qcow2 image and backing file: > > full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2 > > Or the NBD server would be a custom backup application that does > something smart with the incoming NBD WRITE requests. What I care about is connecting libguestfs to qemu and reading a snapshot at some point in time, even though the guest is still writing away to its disks. Is this possible with drive-backup (or otherwise)?

>> What I care about is connecting libguestfs to qemu and reading a >> snapshot at some point in time, even though the guest is still writing >> away to its disks. Is this possible with drive-backup (or otherwise)? > > Yes, that is what drive-backup does. > > New writes coming from the guest are held up until the old data has been > written to the NBD target. > > That way you get a point-in-time snapshot while the guest continues > running. I understand how that can work for backups, where you want to copy a whole disk consistently. But libguestfs doesn't want to do a backup, nor get a copy of the whole disk, it just wants to access a scattering of blocks (maybe a few hundred) but at a single point in time, in as lightweight a manner as possible.

AIUI: We'd issue a drive-backup monitor command with an nbd:... target. The custom NBD server receives a stream of blocks (as writes). On the other side of this, libguestfs is also talking to the custom NBD server. Libguestfs (which is really a qemu process) is issuing random reads. There's no way for the NBD server or anything else to predict what blocks libguestfs will want to read in advance. In the middle of this is our custom NBD server, probably implemented using nbdkit. It has to save all the writes from qemu. It has to satisfy the reads from libguestfs, probably by blocking libguestfs unless we've seen the corresponding write.

The NBD server is going to be (a) storing huge quantities of temporary data which we'll mostly not use, and (b) blocking libguestfs for arbitrary periods of time. This doesn't sound very lightweight to me.

On Fri, Mar 27, 2015 at 03:21:25PM +0000, Stefan Hajnoczi wrote: > On Fri, Mar 27, 2015 at 12:31:41PM +0000, Richard W.M. Jones wrote: > > On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi wrote: > > > On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy wrote: > > > > So, something like? > > > > > > > > . . . > > > > { 'execute': 'drive-backup', 'arguments': > > > > { 'device': 'drive-virtio-disk1', 'sync': 'full', 'target': > > > > 'nbd://localhost', 'mode': 'absolute-paths', 'format': 'qcow2' } > > > > . . . > > > > > > > > Same question as yours, what is the NBD server going to run? > > > > > > > > > > > > My only NBD testing so far has been with w/ NBD over Unix socket or > > > > over TCP[**]. > > > > > > For 'sync': 'full' mode qemu-nbd or nbd-server can be used as the > > > server. You probably don't want 'format': 'qcow2', just raw data over > > > NBD. That way the NBD server can implement whatever storage backend it > > > likes (raw, qcow2, something else). > > > > > > For incremental backup the NBD server would either be a qemu-nbd > > > instance with a qcow2 image and backing file: > > > > > > full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2 > > > > > > Or the NBD server would be a custom backup application that does > > > something smart with the incoming NBD WRITE requests. > > > > What I care about is connecting libguestfs to qemu and reading a > > snapshot at some point in time, even though the guest is still writing > > away to its disks. Is this possible with drive-backup (or otherwise)? > > Yes, that is what drive-backup does. > > New writes coming from the guest are held up until the old data has been > written to the NBD target. > > That way you get a point-in-time snapshot while the guest continues > running. I understand how that can work for backups, where you want to copy a whole disk consistently. But libguestfs doesn't want to do a backup, nor get a copy of the whole disk, it just wants to access a scattering of blocks (maybe a few hundred) but at a single point in time, in as lightweight a manner as possible.