Inspection of disk snapshots
Greetings, I have the following typical scenario: given one or more qcow2 base images I clone them with COW and start the VMs. At a certain point I'd like to inspect them in order to see their evolution compared to the known base images. To do so I was thinking about taking a disk snapshot of each VM and inspect its content through libguestfs (using it's Python bindings). Obviously I need the base image in order for libguestfs to correctly guess the OS, the FS structure etc.. Problem is that that point when I inspect the disk I get the whole disk state including the base image content (30K+ files and directories). This is not an issue but it's a very heavy operation considering that some of the snapshots are few megabytes while the base images are several gigabytes. Is there a way to programmatically instruct libguestfs to limit the inspection to the sole snapshot? Would it work as well with other disk format (vmdk linked clones for example)? For better comprehension I'll show the sequence I'm doing (I might do it wrong of course): virsh -c "qemu:///system" snapshot-create --disk-only <domain-ID> I get the snapshot location from its XML description and then: qemu-img convert -f qcow2 -O qcow2 base_image.qcow2 snapshot.qcow2 At that point I mount it through libguestfs and inspect its content. Thank you.
On Mon, Mar 23, 2015 at 04:34:21PM +0200, NoxDaFox wrote:
Greetings,
I have the following typical scenario: given one or more qcow2 base images I clone them with COW and start the VMs.
At a certain point I'd like to inspect them in order to see their evolution compared to the known base images. To do so I was thinking about taking a disk snapshot of each VM and inspect its content through libguestfs (using it's Python bindings).
Obviously I need the base image in order for libguestfs to correctly guess the OS, the FS structure etc.. Problem is that that point when I inspect the disk I get the whole disk state including the base image content (30K+ files and directories).
This is not an issue but it's a very heavy operation considering that some of the snapshots are few megabytes while the base images are several gigabytes.
Is there a way to programmatically instruct libguestfs to limit the inspection to the sole snapshot? Would it work as well with other disk format (vmdk linked clones for example)?
For better comprehension I'll show the sequence I'm doing (I might do it wrong of course):
virsh -c "qemu:///system" snapshot-create --disk-only <domain-ID>
I get the snapshot location from its XML description and then:
qemu-img convert -f qcow2 -O qcow2 base_image.qcow2 snapshot.qcow2
This makes a copy of the whole disk image. It's also not a consistent (point in time) copy.
At that point I mount it through libguestfs and inspect its content.
As long as you use the 'readonly=1' flag (which is really *essential*, else you'll get disk corruption), you can just point libguestfs at the base image: g = guestfs.GuestFS (python_return_dict=True) g.add_drive_opts ("base_image.qcow2", format="qcow2", readonly=1) That also doesn't get you a consistent snapshot, but it'll work most of the time, and give you a clear error in libguestfs when it doesn't (and won't corrupt your base disk or anything like that, provided you're using readonly=1). The effect of the readonly=1 flag is to create an external snapshot. It is roughly the equivalent of doing: qemu-img create -f qcow2 -b base_image snapshot.qcow2 < point libguestfs at snapshot.qcow2 > If you want lightweight, consistent, point-in-time snapshots (which it sounds like you do), qemu has recently been adding this capability. See the 'drive-backup' monitor command. I've not tried using that and I don't know if it is wired up through libvirt, but libguestfs should be able to consume it since it's just an NBD source. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Mon, Mar 23, 2015 at 10:41:01PM +0000, Richard W.M. Jones wrote:
On Mon, Mar 23, 2015 at 04:34:21PM +0200, NoxDaFox wrote:
Greetings,
I have the following typical scenario: given one or more qcow2 base images I clone them with COW and start the VMs.
At a certain point I'd like to inspect them in order to see their evolution compared to the known base images. To do so I was thinking about taking a disk snapshot of each VM and inspect its content through libguestfs (using it's Python bindings).
Obviously I need the base image in order for libguestfs to correctly guess the OS, the FS structure etc.. Problem is that that point when I inspect the disk I get the whole disk state including the base image content (30K+ files and directories).
This is not an issue but it's a very heavy operation considering that some of the snapshots are few megabytes while the base images are several gigabytes.
Is there a way to programmatically instruct libguestfs to limit the inspection to the sole snapshot? Would it work as well with other disk format (vmdk linked clones for example)?
For better comprehension I'll show the sequence I'm doing (I might do it wrong of course):
virsh -c "qemu:///system" snapshot-create --disk-only <domain-ID>
I get the snapshot location from its XML description and then:
qemu-img convert -f qcow2 -O qcow2 base_image.qcow2 snapshot.qcow2
This makes a copy of the whole disk image. It's also not a consistent (point in time) copy.
Oh I see that you're copying the _snapshot_ that you created with libvirt; it's not a whole disk copy. There's still not any point in doing this, and what I said below stands.
At that point I mount it through libguestfs and inspect its content.
As long as you use the 'readonly=1' flag (which is really *essential*, else you'll get disk corruption), you can just point libguestfs at the base image:
g = guestfs.GuestFS (python_return_dict=True) g.add_drive_opts ("base_image.qcow2", format="qcow2", readonly=1)
That also doesn't get you a consistent snapshot, but it'll work most of the time, and give you a clear error in libguestfs when it doesn't (and won't corrupt your base disk or anything like that, provided you're using readonly=1).
The effect of the readonly=1 flag is to create an external snapshot. It is roughly the equivalent of doing:
qemu-img create -f qcow2 -b base_image snapshot.qcow2 < point libguestfs at snapshot.qcow2 >
If you want lightweight, consistent, point-in-time snapshots (which it sounds like you do), qemu has recently been adding this capability. See the 'drive-backup' monitor command. I've not tried using that and I don't know if it is wired up through libvirt, but libguestfs should be able to consume it since it's just an NBD source.
Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Mon, Mar 23, 2015 at 10:43:30PM +0000, Richard W.M. Jones wrote: [. . .]
This makes a copy of the whole disk image. It's also not a consistent (point in time) copy.
Oh I see that you're copying the _snapshot_ that you created with libvirt; it's not a whole disk copy. There's still not any point in doing this, and what I said below stands.
At that point I mount it through libguestfs and inspect its content.
As long as you use the 'readonly=1' flag (which is really *essential*, else you'll get disk corruption), you can just point libguestfs at the base image:
g = guestfs.GuestFS (python_return_dict=True) g.add_drive_opts ("base_image.qcow2", format="qcow2", readonly=1)
That also doesn't get you a consistent snapshot, but it'll work most of the time, and give you a clear error in libguestfs when it doesn't (and won't corrupt your base disk or anything like that, provided you're using readonly=1).
The effect of the readonly=1 flag is to create an external snapshot. It is roughly the equivalent of doing:
qemu-img create -f qcow2 -b base_image snapshot.qcow2 < point libguestfs at snapshot.qcow2 >
If you want lightweight, consistent, point-in-time snapshots (which it sounds like you do), qemu has recently been adding this capability. See the 'drive-backup' monitor command. I've not tried using that
A small QMP test for 'drive-backup': https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
and I don't know if it is wired up through libvirt,
I don't see it wired up yet in libvirt current `git` master.
but libguestfs should be able to consume it since it's just an NBD source.
-- /kashyap
On Wed, Mar 25, 2015 at 07:38:03PM +0100, Kashyap Chamarthy wrote:
On Mon, Mar 23, 2015 at 10:43:30PM +0000, Richard W.M. Jones wrote:
[. . .]
This makes a copy of the whole disk image. It's also not a consistent (point in time) copy.
Oh I see that you're copying the _snapshot_ that you created with libvirt; it's not a whole disk copy. There's still not any point in doing this, and what I said below stands.
At that point I mount it through libguestfs and inspect its content.
As long as you use the 'readonly=1' flag (which is really *essential*, else you'll get disk corruption), you can just point libguestfs at the base image:
g = guestfs.GuestFS (python_return_dict=True) g.add_drive_opts ("base_image.qcow2", format="qcow2", readonly=1)
That also doesn't get you a consistent snapshot, but it'll work most of the time, and give you a clear error in libguestfs when it doesn't (and won't corrupt your base disk or anything like that, provided you're using readonly=1).
The effect of the readonly=1 flag is to create an external snapshot. It is roughly the equivalent of doing:
qemu-img create -f qcow2 -b base_image snapshot.qcow2 < point libguestfs at snapshot.qcow2 >
If you want lightweight, consistent, point-in-time snapshots (which it sounds like you do), qemu has recently been adding this capability. See the 'drive-backup' monitor command. I've not tried using that
A small QMP test for 'drive-backup':
https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
From what I gather, drive-backup would take a nbd:... target, meaning
That's interesting, but even more interesting would be to see drive-backup writing to NBD. that it is the *client* (not the server). What's interesting is what you're supposed to connect to the server. (Maybe I'm wrong about this however) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
[Added Stefan Hajnoczi, who wrote the 'drive-backup' QMP command and Eric Blake.] On Wed, Mar 25, 2015 at 07:41:23PM +0000, Richard W.M. Jones wrote:
On Wed, Mar 25, 2015 at 07:38:03PM +0100, Kashyap Chamarthy wrote:
On Mon, Mar 23, 2015 at 10:43:30PM +0000, Richard W.M. Jones wrote:
[. . .]
If you want lightweight, consistent, point-in-time snapshots (which it sounds like you do), qemu has recently been adding this capability.
Yeah, I have the two-part incremental backup series from John Snow's (who is the author of it) git repo. But, he said one more revision is about to hit the list, so I held off my testing until I can cleanly test that series on top of current QEMU git master.
See the 'drive-backup' monitor command. I've not tried using that
A small QMP test for 'drive-backup':
https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
That's interesting, but even more interesting would be to see drive-backup writing to NBD.
From what I gather, drive-backup would take a nbd:... target, meaning that it is the *client* (not the server).
You're right -- the original cover letter[*] of the 'drive-backup' QEMU patch series that was merged says it: "The simplest use-case is to copy a point-in-time snapshot to a local file. More advanced users may wish to make the target an NBD URL. The NBD server listening on the other side can process the backup writes any way it wishes."
What's interesting is what you're supposed to connect to the server. (Maybe I'm wrong about this however)
Yeah, I too wonder what to connect on the server side when using NBD as target with 'drive-backup' (taking the simple example test I pointed above). So, something like? . . . { 'execute': 'drive-backup', 'arguments': { 'device': 'drive-virtio-disk1', 'sync': 'full', 'target': 'nbd://localhost', 'mode': 'absolute-paths', 'format': 'qcow2' } . . . Same question as yours, what is the NBD server going to run? My only NBD testing so far has been with w/ NBD over Unix socket or over TCP[**]. [*] http://lists.nongnu.org/archive/html/qemu-devel/2013-06/msg04229.html [**] NBD over TCP: (1) Run an NBD server locally: $ qemu-nbd -f qcow2 -p10809 \ /var/lib/libvirt/images/cirros-0.3.3-x86_64-disk.img -t (2) Create an overlay using the NBD server as backing file: $ qemu-img create -f qcow2 -F raw \ -o backing_file=nbd://localhost nbd-overlay1.qcow2 (3) Optionally, boot the overlay w/ a minimal CLI. $ qemu-system-x86_64 \ -nographic \ -nodefconfig \ -nodefaults \ -m 2048 \ -device virtio-scsi-pci,id=scsi \ -device virtio-serial-pci \ -serial stdio \ -drive file=./nbd-overlay1.qcow2,format=qcow2,if=virtio,cache=writeback -- /kashyap
On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy wrote:
[Added Stefan Hajnoczi, who wrote the 'drive-backup' QMP command and Eric Blake.]
On Wed, Mar 25, 2015 at 07:41:23PM +0000, Richard W.M. Jones wrote:
On Wed, Mar 25, 2015 at 07:38:03PM +0100, Kashyap Chamarthy wrote:
On Mon, Mar 23, 2015 at 10:43:30PM +0000, Richard W.M. Jones wrote:
[. . .]
If you want lightweight, consistent, point-in-time snapshots (which it sounds like you do), qemu has recently been adding this capability.
Yeah, I have the two-part incremental backup series from John Snow's (who is the author of it) git repo. But, he said one more revision is about to hit the list, so I held off my testing until I can cleanly test that series on top of current QEMU git master.
See the 'drive-backup' monitor command. I've not tried using that
A small QMP test for 'drive-backup':
https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
That's interesting, but even more interesting would be to see drive-backup writing to NBD.
From what I gather, drive-backup would take a nbd:... target, meaning that it is the *client* (not the server).
You're right -- the original cover letter[*] of the 'drive-backup' QEMU patch series that was merged says it:
"The simplest use-case is to copy a point-in-time snapshot to a local file.
More advanced users may wish to make the target an NBD URL. The NBD server listening on the other side can process the backup writes any way it wishes."
What's interesting is what you're supposed to connect to the server. (Maybe I'm wrong about this however)
Yeah, I too wonder what to connect on the server side when using NBD as target with 'drive-backup' (taking the simple example test I pointed above).
So, something like?
. . . { 'execute': 'drive-backup', 'arguments': { 'device': 'drive-virtio-disk1', 'sync': 'full', 'target': 'nbd://localhost', 'mode': 'absolute-paths', 'format': 'qcow2' } . . .
Same question as yours, what is the NBD server going to run?
My only NBD testing so far has been with w/ NBD over Unix socket or over TCP[**].
For 'sync': 'full' mode qemu-nbd or nbd-server can be used as the server. You probably don't want 'format': 'qcow2', just raw data over NBD. That way the NBD server can implement whatever storage backend it likes (raw, qcow2, something else). For incremental backup the NBD server would either be a qemu-nbd instance with a qcow2 image and backing file: full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2 Or the NBD server would be a custom backup application that does something smart with the incoming NBD WRITE requests. Stefan
[*] http://lists.nongnu.org/archive/html/qemu-devel/2013-06/msg04229.html
[**] NBD over TCP:
(1) Run an NBD server locally:
$ qemu-nbd -f qcow2 -p10809 \ /var/lib/libvirt/images/cirros-0.3.3-x86_64-disk.img -t
(2) Create an overlay using the NBD server as backing file:
$ qemu-img create -f qcow2 -F raw \ -o backing_file=nbd://localhost nbd-overlay1.qcow2
(3) Optionally, boot the overlay w/ a minimal CLI.
$ qemu-system-x86_64 \ -nographic \ -nodefconfig \ -nodefaults \ -m 2048 \ -device virtio-scsi-pci,id=scsi \ -device virtio-serial-pci \ -serial stdio \ -drive file=./nbd-overlay1.qcow2,format=qcow2,if=virtio,cache=writeback
-- /kashyap
On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi wrote:
On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy wrote:
So, something like?
. . . { 'execute': 'drive-backup', 'arguments': { 'device': 'drive-virtio-disk1', 'sync': 'full', 'target': 'nbd://localhost', 'mode': 'absolute-paths', 'format': 'qcow2' } . . .
Same question as yours, what is the NBD server going to run?
My only NBD testing so far has been with w/ NBD over Unix socket or over TCP[**].
For 'sync': 'full' mode qemu-nbd or nbd-server can be used as the server. You probably don't want 'format': 'qcow2', just raw data over NBD. That way the NBD server can implement whatever storage backend it likes (raw, qcow2, something else).
For incremental backup the NBD server would either be a qemu-nbd instance with a qcow2 image and backing file:
full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2
Or the NBD server would be a custom backup application that does something smart with the incoming NBD WRITE requests.
What I care about is connecting libguestfs to qemu and reading a snapshot at some point in time, even though the guest is still writing away to its disks. Is this possible with drive-backup (or otherwise)? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Fri, Mar 27, 2015 at 12:31:41PM +0000, Richard W.M. Jones wrote:
On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi wrote:
On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy wrote:
So, something like?
. . . { 'execute': 'drive-backup', 'arguments': { 'device': 'drive-virtio-disk1', 'sync': 'full', 'target': 'nbd://localhost', 'mode': 'absolute-paths', 'format': 'qcow2' } . . .
Same question as yours, what is the NBD server going to run?
My only NBD testing so far has been with w/ NBD over Unix socket or over TCP[**].
For 'sync': 'full' mode qemu-nbd or nbd-server can be used as the server. You probably don't want 'format': 'qcow2', just raw data over NBD. That way the NBD server can implement whatever storage backend it likes (raw, qcow2, something else).
For incremental backup the NBD server would either be a qemu-nbd instance with a qcow2 image and backing file:
full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2
Or the NBD server would be a custom backup application that does something smart with the incoming NBD WRITE requests.
What I care about is connecting libguestfs to qemu and reading a snapshot at some point in time, even though the guest is still writing away to its disks. Is this possible with drive-backup (or otherwise)?
Yes, that is what drive-backup does. New writes coming from the guest are held up until the old data has been written to the NBD target. That way you get a point-in-time snapshot while the guest continues running. Stefan
On Fri, Mar 27, 2015 at 03:21:25PM +0000, Stefan Hajnoczi wrote:
On Fri, Mar 27, 2015 at 12:31:41PM +0000, Richard W.M. Jones wrote:
On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi wrote:
On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy wrote:
So, something like?
. . . { 'execute': 'drive-backup', 'arguments': { 'device': 'drive-virtio-disk1', 'sync': 'full', 'target': 'nbd://localhost', 'mode': 'absolute-paths', 'format': 'qcow2' } . . .
Same question as yours, what is the NBD server going to run?
My only NBD testing so far has been with w/ NBD over Unix socket or over TCP[**].
For 'sync': 'full' mode qemu-nbd or nbd-server can be used as the server. You probably don't want 'format': 'qcow2', just raw data over NBD. That way the NBD server can implement whatever storage backend it likes (raw, qcow2, something else).
For incremental backup the NBD server would either be a qemu-nbd instance with a qcow2 image and backing file:
full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2
Or the NBD server would be a custom backup application that does something smart with the incoming NBD WRITE requests.
What I care about is connecting libguestfs to qemu and reading a snapshot at some point in time, even though the guest is still writing away to its disks. Is this possible with drive-backup (or otherwise)?
Yes, that is what drive-backup does.
New writes coming from the guest are held up until the old data has been written to the NBD target.
That way you get a point-in-time snapshot while the guest continues running.
I understand how that can work for backups, where you want to copy a whole disk consistently. But libguestfs doesn't want to do a backup, nor get a copy of the whole disk, it just wants to access a scattering of blocks (maybe a few hundred) but at a single point in time, in as lightweight a manner as possible. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On 03/27/2015 09:35 AM, Richard W.M. Jones wrote:
What I care about is connecting libguestfs to qemu and reading a snapshot at some point in time, even though the guest is still writing away to its disks. Is this possible with drive-backup (or otherwise)?
Yes, that is what drive-backup does.
New writes coming from the guest are held up until the old data has been written to the NBD target.
That way you get a point-in-time snapshot while the guest continues running.
I understand how that can work for backups, where you want to copy a whole disk consistently.
But libguestfs doesn't want to do a backup, nor get a copy of the whole disk, it just wants to access a scattering of blocks (maybe a few hundred) but at a single point in time, in as lightweight a manner as possible.
If you KNOW what sectors you want to read, then your NBD target can ignore writes to the sectors you don't care about (the guest is changing data on a sector you don't care about; yeah, bandwidth was spent in telling you that, but you don't have to spend storage on it), while focusing on reading the sectors you do care about as fast as possible (to minimize the time spent dealing with uninteresting writes). If you DON'T know what sectors you want to read (because you are chasing file system pointers and don't know a priori where those pointers will resolve), then tracking ALL data flushed by guest writes IS the most efficient manner for keeping your point in time accurate for the duration of your fleecing operation. Either way, if you really are going to read only a few hundred sectors and then close the connection, it shouldn't matter if the drive-backup failed to send all guest sectors modified after the point in time, so long as every sector you read is accurate (either because the guest hasn't touched it in the meantime; or because even though the guest touched it after the point in time, you were given the original contents through your NBD target). -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Fri, Mar 27, 2015 at 10:37:44AM -0600, Eric Blake wrote:
On 03/27/2015 09:35 AM, Richard W.M. Jones wrote:
But libguestfs doesn't want to do a backup, nor get a copy of the whole disk, it just wants to access a scattering of blocks (maybe a few hundred) but at a single point in time, in as lightweight a manner as possible.
If you KNOW what sectors you want to read, then your NBD target can ignore writes to the sectors you don't care about (the guest is changing data on a sector you don't care about; yeah, bandwidth was spent in telling you that, but you don't have to spend storage on it), while focusing on reading the sectors you do care about as fast as possible (to minimize the time spent dealing with uninteresting writes). If you DON'T know what sectors you want to read (because you are chasing file system pointers and don't know a priori where those pointers will resolve), then tracking ALL data flushed by guest writes IS the most efficient manner for keeping your point in time accurate for the duration of your fleecing operation. Either way, if you really are going to read only a few hundred sectors and then close the connection, it shouldn't matter if the drive-backup failed to send all guest sectors modified after the point in time, so long as every sector you read is accurate (either because the guest hasn't touched it in the meantime; or because even though the guest touched it after the point in time, you were given the original contents through your NBD target).
AIUI: We'd issue a drive-backup monitor command with an nbd:... target. The custom NBD server receives a stream of blocks (as writes). On the other side of this, libguestfs is also talking to the custom NBD server. Libguestfs (which is really a qemu process) is issuing random reads. There's no way for the NBD server or anything else to predict what blocks libguestfs will want to read in advance. In the middle of this is our custom NBD server, probably implemented using nbdkit. It has to save all the writes from qemu. It has to satisfy the reads from libguestfs, probably by blocking libguestfs unless we've seen the corresponding write. The NBD server is going to be (a) storing huge quantities of temporary data which we'll mostly not use, and (b) blocking libguestfs for arbitrary periods of time. This doesn't sound very lightweight to me. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On 03/27/2015 11:21 AM, Richard W.M. Jones wrote:
AIUI:
We'd issue a drive-backup monitor command with an nbd:... target.
The custom NBD server receives a stream of blocks (as writes).
On the other side of this, libguestfs is also talking to the custom NBD server. Libguestfs (which is really a qemu process) is issuing random reads. There's no way for the NBD server or anything else to predict what blocks libguestfs will want to read in advance.
In the middle of this is our custom NBD server, probably implemented using nbdkit. It has to save all the writes from qemu. It has to satisfy the reads from libguestfs, probably by blocking libguestfs unless we've seen the corresponding write.
Well, it only has to write the sectors touched by the guest in the meantime, not the entire disk. But yeah, a busy guest can cause a lot of sectors to be written in the meantime.
The NBD server is going to be (a) storing huge quantities of temporary data which we'll mostly not use, and (b) blocking libguestfs for arbitrary periods of time. This doesn't sound very lightweight to me.
Hmm. Sounds a bit like we want to take advantage of postcopy migration smarts - where the destination receives the full stream of writes as a low-priority, but can interject and request out-of-order reads to satisfy page faults on a high-priority. All reads are guaranteed to resolve to the correct data, even if it means blocking the read until the out-of-order page fault is read in, but the out-of-order processing means that you don't have to wait for the full stream to take place before you get the information you need a the moment. Is NBD bi-directional, in that the target can receive write requests at the same time it is sending read requests? It sounds like that is what we need. Or are we stuck with NBD being uni-directional, where the target can receive read and write commands, but can't send any commands back to the client in charge of the data being written? If that's the case, maybe the work in qemu 2.4 towards persistent dirty bitmaps can help: set up a bitmap before starting the NBD server, to track what the guest is dirtying. With the bitmap in place, then for every sector you want, you first read it directly from the source image, THEN check the persistent dirty bitmap to see if the sector has been marked for transfer to NBD. If so, then you'll have to wait for it to show up on the NBD target side; if not, then the guest hasn't touched it yet so you know what you read is correct. That still doesn't help optimizing out the writes to the NBD target for sectors you don't care about, and doesn't quite address the desire for making random reads take priority over linear streaming of dirty blocks, but it might help. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Fri, Mar 27, 2015 at 01:27:39PM -0600, Eric Blake wrote:
Is NBD bi-directional, in that the target can receive write requests at the same time it is sending read requests?
It's unidirectional. It's a very simple protocol - you can't even answer requests out of order: https://github.com/yoe/nbd/blob/master/doc/proto.txt In an ideal world for libguestfs we'd have qemu be the NBD server, with some sort of monitor command like: drive-point-in-time-snapshot device:virtio-disk1 port:1000 id:123 then you'd connect to <qemu-server>:1000 and it'd give you access to the drive at that instant in time until there was a further command to discard the snapshot: drive-discard-point-in-time-snapshot id:123 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Fri, Mar 27, 2015 at 03:35:02PM +0000, Richard W.M. Jones wrote:
On Fri, Mar 27, 2015 at 03:21:25PM +0000, Stefan Hajnoczi wrote:
On Fri, Mar 27, 2015 at 12:31:41PM +0000, Richard W.M. Jones wrote:
On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi wrote:
On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy wrote:
So, something like?
. . . { 'execute': 'drive-backup', 'arguments': { 'device': 'drive-virtio-disk1', 'sync': 'full', 'target': 'nbd://localhost', 'mode': 'absolute-paths', 'format': 'qcow2' } . . .
Same question as yours, what is the NBD server going to run?
My only NBD testing so far has been with w/ NBD over Unix socket or over TCP[**].
For 'sync': 'full' mode qemu-nbd or nbd-server can be used as the server. You probably don't want 'format': 'qcow2', just raw data over NBD. That way the NBD server can implement whatever storage backend it likes (raw, qcow2, something else).
For incremental backup the NBD server would either be a qemu-nbd instance with a qcow2 image and backing file:
full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2
Or the NBD server would be a custom backup application that does something smart with the incoming NBD WRITE requests.
What I care about is connecting libguestfs to qemu and reading a snapshot at some point in time, even though the guest is still writing away to its disks. Is this possible with drive-backup (or otherwise)?
Yes, that is what drive-backup does.
New writes coming from the guest are held up until the old data has been written to the NBD target.
That way you get a point-in-time snapshot while the guest continues running.
I understand how that can work for backups, where you want to copy a whole disk consistently.
But libguestfs doesn't want to do a backup, nor get a copy of the whole disk, it just wants to access a scattering of blocks (maybe a few hundred) but at a single point in time, in as lightweight a manner as possible.
There is a way to do that. Create a temporary qcow2 backed by the disk: guest.img <- tmp.qcow2 Now use drive-backup with sync mode 'none' so only new guest writes cause old data to be stashed in tmp.qcow2 before it is overwritten. Finally, use the run-time NBD server for random access to tmp.qcow2. When you are done, kill the drive-backup job and delete tmp.qcow2. Stefan
participants (5)
-
Eric Blake -
Kashyap Chamarthy -
NoxDaFox -
Richard W.M. Jones -
Stefan Hajnoczi