12:11 p.m.
Richard-
>> Per this page
>>
>> https://rwmj.wordpress.com/2013/04/22/using-rsync-with-libguestfs/
>>
>> We are trying to get the libguestfs API guestfs_rsync_out to work. Weve tried
combinations similar to:
>>
>> guestfs_rsync_out(g, /HostShared, rsync://root@10.0.1.90:2999/backup/ -av,
-1)
>>
>> surrounding it with APIs as in the recipe examples, for example replacing
libguestfs_cat(), with no luck. In the
>> above
>> example the "backup" rsync module name matches the script on
Richard's wordpress.com blog page (above link).
>> Typically in the rsync daemon log we see something like:
>>
>> receiving file list
>> sent 17 bytes received 24 bytes total size 0
>>
>> but no data is transferred. We are successful using rsync from the Linux command
line, and we can get libguestfs
>> APIs
>> like guestfs_cat() to work fine. But weve had zero luck with rsync_out over the
last week.
>
> I think it's really hard to diagnose problems without turning on
> debugging in libguestfs:
>
> http://libguestfs.org/guestfs-faq.1.html#how-do-i-debug-when-using-the-api
>
>> Unfortunately there are no actual API examples online, no
>> recipes on the libguestfs.org site, etc. Does anyone know
>> where there are some actual libguestfs_rsync_out() working examples?
>
> The only example I can point to is this test:
>
> https://github.com/libguestfs/libguestfs/blob/master/tests/rsync/test-rsy...
Thanks very much. In our example, the libguestfs API syntax required turns out to be:
guestfs_rsync_out(g, "/Win7Folder",
"rsync://root@10.0.1.90:2999/backup", GUESTFS_RSYNC_OUT_ARCHIVE, 1, -1)
This produces a trace result:
libguestfs: trace: rsync_out "/Win7Folder"
"rsync://root@10.0.1.90:2999/backup" "archive:true"
Also, to avoid chgrp and other permission related messages, we had to add:
uid = 0
gid = 0
in our rsyncd.conf file (we are running rsync daemaon as root).
Now we are testing guestfs_rsync_out() in a loop that runs every 1 sec. We hope to
achieve continuous synchronization
with a live Win7 guest. As I mentioned before, we're only reading from the Win7
guest, and not a lot of data is
involved, maybe a few files up to 50 MByte. These files are not changing size, just
contents.
[Update]
With continuous loop testing, what we found is that we have to shut down and re-launch the
image handle to see changes
on the Win7 live guest. Unfortunately image re-launch takes time, 3-5 sec (the image size
is 50 GByte). I'm assuming
this is because libguestfs makes an internal copy of filesystem, and works from that, and
doesn't "refresh" this
internal copy until re-launching the image handle.
Could we create a second partition on the guest that is much smaller, say 50 MByte, attach
the image to that, and thus
reduce the shutdown and re-launch time into the msec range? Is there another real-time
method, not using rsync? As a
note, for Linux guest images, we use virtFs, but it seems that WinX doesn't support
the 9p filesystem drivers required
for virtFs.
Thanks.
-Jeff
2:59 a.m.
New subject: getting guestfs_rsync_out to work
On Sun, Feb 08, 2015 at 12:11:37PM -0600, Jeff Brower wrote:
With continuous loop testing, what we found is that we have to shut
down and re-launch the image handle to see changes on the Win7 live
guest. Unfortunately image re-launch takes time, 3-5 sec (the image
size is 50 GByte). I'm assuming this is because libguestfs makes an
internal copy of filesystem, and works from that, and doesn't
"refresh" this internal copy until re-launching the image handle.
This isn't surprising. Libguestfs doesn't make an internal copy of
the whole filesystem, but the libguestfs appliance will have a copy
(in its kernel memory) of any parts of the disk that rsync read.
See also:
http://libguestfs.org/guestfs.3.html#architecture
Could we create a second partition on the guest that is much
smaller, say 50 MByte, attach the image to that, and thus reduce the
shutdown and re-launch time into the msec range? Is there another
real-time method, not using rsync?
The time to relaunch the appliance doesn't depend on the size of the
disk.
You could try hotplugging but it may not be much faster:
http://libguestfs.org/guestfs.3.html#hotplugging
I think what you really need to do is to install a backup agent in the
Windows guest and use a regular network backup.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
8:01 a.m.
New subject: getting guestfs_rsync_out to work
Richard-
On Sun, Feb 08, 2015 at 12:11:37PM -0600, Jeff Brower wrote:
> With continuous loop testing, what we found is that we have to shut
> down and re-launch the image handle to see changes on the Win7 live
> guest. Unfortunately image re-launch takes time, 3-5 sec (the image
> size is 50 GByte). I'm assuming this is because libguestfs makes an
> internal copy of filesystem, and works from that, and doesn't
> "refresh" this internal copy until re-launching the image handle.
This isn't surprising. Libguestfs doesn't make an internal copy of
the whole filesystem, but the libguestfs appliance will have a copy
(in its kernel memory) of any parts of the disk that rsync read.
See also:
http://libguestfs.org/guestfs.3.html#architecture
> Could we create a second partition on the guest that is much
> smaller, say 50 MByte, attach the image to that, and thus reduce the
> shutdown and re-launch time into the msec range? Is there another
> real-time method, not using rsync?
The time to relaunch the appliance doesn't depend on the size of the
disk.
You could try hotplugging but it may not be much faster:
http://libguestfs.org/guestfs.3.html#hotplugging
I think what you really need to do is to install a backup agent in the
Windows guest and use a regular network backup.
We don't need to use rsync. We know which files will change. For example even if we
do this (error checking removed):
while (1) {
guestfs_mount_options(g, "sync", "/dev/sda2", "/");
printf("%s\n", guestfs_cat(g, "/HostShared/temp.txt"));
guestfs_umount(g, "/");
guestfs_fsync(g);
usleep(1000*1000);
}
temp.txt from the Win7 guest is copied correctly the first time, but afterwards any change
we make to it (e.g. using
Win7 Notepad) is never seen. That's the issue. Is there any way at all to accomplish
this using libguestfs, without
re-launching the image, which is time-consuming?
Thanks.
-Jeff
8:23 a.m.
New subject: getting guestfs_rsync_out to work
On Mon, Feb 09, 2015 at 08:01:31AM -0600, Jeff Brower wrote:
Richard-
> On Sun, Feb 08, 2015 at 12:11:37PM -0600, Jeff Brower wrote:
>> With continuous loop testing, what we found is that we have to shut
>> down and re-launch the image handle to see changes on the Win7 live
>> guest. Unfortunately image re-launch takes time, 3-5 sec (the image
>> size is 50 GByte). I'm assuming this is because libguestfs makes an
>> internal copy of filesystem, and works from that, and doesn't
>> "refresh" this internal copy until re-launching the image handle.
>
> This isn't surprising. Libguestfs doesn't make an internal copy of
> the whole filesystem, but the libguestfs appliance will have a copy
> (in its kernel memory) of any parts of the disk that rsync read.
>
> See also:
> http://libguestfs.org/guestfs.3.html#architecture
>
>> Could we create a second partition on the guest that is much
>> smaller, say 50 MByte, attach the image to that, and thus reduce the
>> shutdown and re-launch time into the msec range? Is there another
>> real-time method, not using rsync?
>
> The time to relaunch the appliance doesn't depend on the size of the
> disk.
>
> You could try hotplugging but it may not be much faster:
>
> http://libguestfs.org/guestfs.3.html#hotplugging
>
> I think what you really need to do is to install a backup agent in the
> Windows guest and use a regular network backup.
We don't need to use rsync. We know which files will change. For example even if we
do this (error checking removed):
while (1) {
guestfs_mount_options(g, "sync", "/dev/sda2", "/");
printf("%s\n", guestfs_cat(g, "/HostShared/temp.txt"));
guestfs_umount(g, "/");
guestfs_fsync(g);
usleep(1000*1000);
}
temp.txt from the Win7 guest is copied correctly the first time, but
afterwards any change we make to it (e.g. using Win7 Notepad) is
never seen. That's the issue.
This is completely expected. Have a look at the architecture diagram
here:
http://libguestfs.org/guestfs.3.html#architecture
Now imagine there is another line drawn from the "Device or disk
image" box to your running Windows guest. How does the appliance know
that there's something else making changes to the disk?
BTW you really *must* be using the readonly flag when adding disks
(guestfs_add_drive_opts) else you will get disk corruption.
Is there any way at all to accomplish this using libguestfs, without
re-launching the image, which is time-consuming?
You could try calling `guestfs_drop_caches (g, 3);' (which may or may
not work depending on a bunch of details in how Linux and qemu works),
or using hotplugging.
However you need to understand how libguestfs, the kernel and qemu are
architected and why the naive approach is failing first.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
8:27 a.m.
New subject: getting guestfs_rsync_out to work
Richard-
On Mon, Feb 09, 2015 at 08:01:31AM -0600, Jeff Brower wrote:
> Richard-
>
> > On Sun, Feb 08, 2015 at 12:11:37PM -0600, Jeff Brower wrote:
> >> With continuous loop testing, what we found is that we have to shut
> >> down and re-launch the image handle to see changes on the Win7 live
> >> guest. Unfortunately image re-launch takes time, 3-5 sec (the image
> >> size is 50 GByte). I'm assuming this is because libguestfs makes an
> >> internal copy of filesystem, and works from that, and doesn't
> >> "refresh" this internal copy until re-launching the image handle.
> >
> > This isn't surprising. Libguestfs doesn't make an internal copy of
> > the whole filesystem, but the libguestfs appliance will have a copy
> > (in its kernel memory) of any parts of the disk that rsync read.
> >
> > See also:
> > http://libguestfs.org/guestfs.3.html#architecture
> >
> >> Could we create a second partition on the guest that is much
> >> smaller, say 50 MByte, attach the image to that, and thus reduce the
> >> shutdown and re-launch time into the msec range? Is there another
> >> real-time method, not using rsync?
> >
> > The time to relaunch the appliance doesn't depend on the size of the
> > disk.
> >
> > You could try hotplugging but it may not be much faster:
> >
> > http://libguestfs.org/guestfs.3.html#hotplugging
> >
> > I think what you really need to do is to install a backup agent in the
> > Windows guest and use a regular network backup.
>
> We don't need to use rsync. We know which files will change. For example even
if we do this (error checking
> removed):
>
> while (1) {
>
> guestfs_mount_options(g, "sync", "/dev/sda2",
"/");
>
> printf("%s\n", guestfs_cat(g, "/HostShared/temp.txt"));
>
> guestfs_umount(g, "/");
>
> guestfs_fsync(g);
>
> usleep(1000*1000);
> }
>
> temp.txt from the Win7 guest is copied correctly the first time, but
> afterwards any change we make to it (e.g. using Win7 Notepad) is
> never seen. That's the issue.
This is completely expected. Have a look at the architecture diagram
here:
http://libguestfs.org/guestfs.3.html#architecture
Now imagine there is another line drawn from the "Device or disk
image" box to your running Windows guest. How does the appliance know
that there's something else making changes to the disk?
Yes we know that it doesn't know. We'd be ok to re-launch the image to pick up
changes except for the time it takes
(if we could get into 1 to 10 msec range it would be ok).
BTW you really *must* be using the readonly flag when adding disks
(guestfs_add_drive_opts) else you will get disk corruption.
Yes we have been doing that.
> Is there any way at all to accomplish this using libguestfs,
without
> re-launching the image, which is time-consuming?
You could try calling `guestfs_drop_caches (g, 3);' (which may or may
not work depending on a bunch of details in how Linux and qemu works),
or using hotplugging.
Ok we'll try that. Thank you for the suggestion.
-Jeff
9:14 a.m.
New subject: getting guestfs_rsync_out to work
On Mon, Feb 09, 2015 at 08:27:22AM -0600, Jeff Brower wrote:
Yes we know that it doesn't know. We'd be ok to re-launch
the image
to pick up changes except for the time it takes (if we could get
into 1 to 10 msec range it would be ok).
There's no feasible way to start a Linux kernel in under about 1-2
seconds. It just takes that long for the kernel to initialize itself.
Even if you use the User-mode Linux backend (which just runs vmlinux
as a userspace process) you'll experience approximately the same
overhead.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
3:39 p.m.
New subject: getting guestfs_rsync_out to work
Richard-
On Mon, Feb 09, 2015 at 08:27:22AM -0600, Jeff Brower wrote:
> Yes we know that it doesn't know. We'd be ok to re-launch the image
> to pick up changes except for the time it takes (if we could get
> into 1 to 10 msec range it would be ok).
There's no feasible way to start a Linux kernel in under about 1-2
seconds. It just takes that long for the kernel to initialize itself.
Even if you use the User-mode Linux backend (which just runs vmlinux
as a userspace process) you'll experience approximately the same
overhead.
Ok got it. Thanks again for your reply. We tried guestfs_drop_caches() as follows:
while(1) {
guestfs_mount(g, "/dev/sda2", "/");
printf("%s\n", guestfs_cat(g, "/HostShared/temp.txt"));
guestfs_umount(g, "/");
guestfs_drop_caches(g, 3);
usleep(1000*1000);
}
and still we cannot see changes made to temp.txt (from inside the Win7 guest VM)
while this loop continues to run on the host Linux. We still have to re-launch the
image to see changes.
In general, it would be great if the "kernel load" and "disk refresh /
sync"
functions in guestfs_launch() could be separated. If there was a way to do read-only
file sharing, a very limited subset of what virtFs/9p can do, that would be great for
Win guests.
-Jeff
8:05 a.m.
New subject: Enable Smart Array P212 thru VM
Hi,
I'd like to know if there is any solution to enable a Smart Array P212 PCIe card over
a Windows VM ?
I've a Backup Acronis Software to install, and I'd prefer to install it on a
CentOS distrib, and not on a Windows 2008 R2.
Our old backup software is Symantec Backup Exec and we have to keep the possibility to
restore old tapes on demand.
So the solution to keep a sleeping Windows VM, and have the way to enable it, just to
restore a tape would be perfect.
I've tried to add a new hardware with virt-manager, but this action just freeze the
manager, and I need to restart the libvirtd service to unlock it.
If anybody knows a way to do that, I'd be very grateful.
Thanks in advance
Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76
UNICEF France
3 rue Duguay Trouin 75006 PARIS
www.unicef.fr
3497
days inactive
3499
days old
7 comments
3 participants
participants (3)
-
Jeff Brower -
Richard W.M. Jones -
VONDRA Alain