ntfs-3g data deduplication support in guestmount for creating file based back-ups on visualization platform
Hello everybody, I am using libguestfs-tools version 1.28.1 in a bunch of bash scripts to make back-ups of the files on a visualisation platform. I been trying to get read access to ntfs volumes with data deduplication working. The ntfs-3g guys developed some support and I have this working when using the commands our back-up servers. However when using the guestmount tool the read suport foor dedub is gone. See this email for all the commands I use (working mount -t ntfs and non-working guestmount commands). https://sourceforge.net/p/ntfs-3g/mailman/message/35595392/ How can I see what version of ntfs-3g is used in guestmount and if it is using the /usr/lib/x86_64-linux-gnu/ntfs-3g/ntfs-plugin-80000013.so plugin that is on my back-up server? Thank you in advance! Kind regards, Jelle de Jong
On Tue, Jan 10, 2017 at 05:06:13PM +0100, Jelle de Jong wrote:
Hello everybody,
I am using libguestfs-tools version 1.28.1 in a bunch of bash scripts to make back-ups of the files on a visualisation platform.
I been trying to get read access to ntfs volumes with data deduplication working.
Do you have more information about data deduplication? It's the first I've heard of it for NTFS (or indeed any ordinary Unix filesystem). In fact I thought it was a feature of the block layer rather than the filesystem. But anyway ...
The ntfs-3g guys developed some support and I have this working when using the commands our back-up servers.
However when using the guestmount tool the read suport foor dedub is gone.
This is maybe not especially surprising. A lot of layers have to work correctly in order for a feature to be passed through guestmount. You've got (at least) kernel+FUSE on the host side (for guestmount), the libguestfs API, the appliance kernel, and ntfs-3g in the appliance. Getting discard support working involved a considerable effort.
See this email for all the commands I use (working mount -t ntfs and non-working guestmount commands).
https://sourceforge.net/p/ntfs-3g/mailman/message/35595392/
How can I see what version of ntfs-3g is used in guestmount and if it is using the /usr/lib/x86_64-linux-gnu/ntfs-3g/ntfs-plugin-80000013.so plugin that is on my back-up server?
Probably helpful to refer to the diagram here: http://libguestfs.org/guestfs-internals.1.html ntfs-3g in your case is running in the "appliance" box in that diagram. The plugin file *should* be copied into the appliance, assuming that it's part of the Debian ntfs-3g package and the Debian ntfs-3g package isn't doing anything weird. You can easily check using virt-rescue: $ virt-rescue --ro -a windows.img
<rescue> ls -l /usr/lib/x86_64-linux-gnu/ntfs-3g/ <rescue> mount /dev/sda1 /sysroot
(and then use whatever ntfs commands are needed to check that the plugin is being used) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
Thank you Richard! I will do some more work and testing on this tomorrow, and report back with some more information you asked. In the meanwhile, could you help me with information how I can upgrade the ntfs-3g version (same as host) in the "appliance" and add the plug-in library? On 16/01/17 15:29, Richard W.M. Jones wrote:
The plugin file *should* be copied into the appliance, assuming that it's part of the Debian ntfs-3g package and the Debian ntfs-3g package isn't doing anything weird.
The plug-in is added manually and is not part of the ntfs-3g (yet), it is manually downloaded and copied. Thank you in advance! Kind regards, Jelle de Jong
On Mon, Jan 16, 2017 at 04:20:14PM +0100, Jelle de Jong wrote:
Thank you Richard!
I will do some more work and testing on this tomorrow, and report back with some more information you asked.
In the meanwhile, could you help me with information how I can upgrade the ntfs-3g version (same as host) in the "appliance" and add the plug-in library? [...] The plug-in is added manually and is not part of the ntfs-3g (yet), it is manually downloaded and copied.
You can most likely drop a file into /usr/lib/guestfs/supermin.d/ which contains a single line, being the full path of the plugin file that you want to add. Actually you can use wildcards there too, so this should work: echo '/usr/lib/x86_64-linux-gnu/ntfs-3g/ntfs-plugin-*.so' > /usr/lib/guestfs/supermin.d/zz-ntfs-plugins After that, do: rm -rf /var/tmp/.guestfs-* and rerun your libguestfs program (guestmount or whatever). See "SUPERMIN APPLIANCES" section in the supermin(1) man page for further information. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Hi Richard, Is there a way I can access the /var/log/syslog of an running guestmount instance to get the grep ntfs-3g /var/log/syslog information? I managed to get the plug-in to load in my questmount instances by using: echo '/usr/lib/x86_64-linux-gnu/ntfs-3g/ntfs-plugin-*.so' > /usr/lib/guestfs/supermin.d/zz-ntfs-plugins rm -rf /var/tmp/.guestfs-* Kind regards, Jelle de Jong On 16/01/17 16:43, Richard W.M. Jones wrote:
On Mon, Jan 16, 2017 at 04:20:14PM +0100, Jelle de Jong wrote:
Thank you Richard!
I will do some more work and testing on this tomorrow, and report back with some more information you asked.
In the meanwhile, could you help me with information how I can upgrade the ntfs-3g version (same as host) in the "appliance" and add the plug-in library? [...] The plug-in is added manually and is not part of the ntfs-3g (yet), it is manually downloaded and copied.
You can most likely drop a file into /usr/lib/guestfs/supermin.d/ which contains a single line, being the full path of the plugin file that you want to add.
Actually you can use wildcards there too, so this should work:
echo '/usr/lib/x86_64-linux-gnu/ntfs-3g/ntfs-plugin-*.so' > /usr/lib/guestfs/supermin.d/zz-ntfs-plugins
After that, do:
rm -rf /var/tmp/.guestfs-*
and rerun your libguestfs program (guestmount or whatever).
See "SUPERMIN APPLIANCES" section in the supermin(1) man page for further information.
Rich.
On Thu, Feb 09, 2017 at 10:05:36PM +0100, Jelle de Jong wrote:
Hi Richard,
Is there a way I can access the /var/log/syslog of an running guestmount instance to get the grep ntfs-3g /var/log/syslog information?
Not of a running guestmount, no, however if you're using the API then you can use the debug "back door" to read dmesg and log files. There is no documentation for it, but you can find out what is available by browsing the sources: https://github.com/libguestfs/libguestfs/blob/master/daemon/debug.c#L83 Something like: const char *cmd[] = { "cat", "/var/log/syslog", NULL }; guestfs_debug (g, "sh", (char**)cmd); or the equivalent in other language bindings. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
participants (2)
-
Jelle de Jong -
Richard W.M. Jones