[PATCH 0/2] added icat and fls0 APIs for deleted files recovery
As discussed in the topic: https://www.redhat.com/archives/libguestfs/2016-March/msg00018.html I'd like to add to libguestfs the disk forensics capabilities offered by The Sleuth Kit. http://www.sleuthkit.org/ The two APIs I'm adding with the patch are a simple example of which type of features TSK can enable. The icat API it's very similar to the previously added ntfscat_i but it works with multiple filesystem types and has an interesting added value: it can extract deleted files. The end result heavily depends on the filesystem internals and on how much it has been used. On FAT, Ext2 and NTFS it can recover part or the whole file pretty easily as long as it's sectors have not been overwritten by others. For Ext3 and Ext4 it's a bit more difficult, usually data carving is more effective in these cases. I haven't tested on other filesystems. The fls0 API works similar to find but it returns more information such as the files/directories metadata addresses (inodes) and whether the files are allocated (in use) or not (deleted). Later on I will add more fine grained APIs such as fls0-d for showing only deleted files and fls0-m which reports the creation/access/deletion timestamps. I might need some help for the related fls API not relying on an external file. A concrete example on a Windows 7 disk image. $ ./run guestfish --ro -a /home/noxdafox/disks/win7clone.qcow2
<fs> run
There's no need to mount the disks as TSK reads the raw data structures of the filesystems.
<fs> fls0 /dev/sda2 /home/noxdafox/disk-content.txt
r/r 15711-128-1: $Recycle.Bin/S-1-5-21-2379395878-2832339042-1309242031-1000/desktop.ini -/r * 60015-128-1: $Recycle.Bin/S-1-5-21-2379395878-2832339042-1309242031-1000/$R07QQZ2.txt -/r * 60015-128-3: $Recycle.Bin/S-1-5-21-2379395878-2832339042-1309242031-1000/$R07QQZ2.txt:Zone.Identifier We can see the content of the recycle bin where I trashed and deleted a readme.txt file (renamed as $R07QQZ2.txt). The asterisk (*) shows the file is not allocated and it has been deleted from the disk. We can see its inode number (60015) which means the inode has not been recycled yet so there's quite a good chance we can recover it.
<fs> icat /dev/sda2 60015 /home/noxdafox/recovered.txt
The file gets fully recovered in my home directory. The correct sha1 proves the recover was 100% successful. I decided to add these APIs because they cover a use case I think libguestfs could immediately benefit from. With TSK tools is possible to do more in depth disk analysis to quickly discover hiddend partitions (malware rootkits) or hidded data in unallocated sectors. Matteo Cafasso (2): added icat and fls0 APIs added icat and fls0 APIs tests Makefile.am | 1 + appliance/packagelist.in | 3 ++ configure.ac | 1 + daemon/Makefile.am | 1 + daemon/tsk.c | 122 +++++++++++++++++++++++++++++++++++++++++++++++ generator/actions.ml | 33 +++++++++++++ po/POTFILES | 1 + src/MAX_PROC_NR | 2 +- tests/tsk/Makefile.am | 27 +++++++++++ tests/tsk/test-fls0.sh | 55 +++++++++++++++++++++ tests/tsk/test-icat.sh | 53 ++++++++++++++++++++ 11 files changed, 298 insertions(+), 1 deletion(-) create mode 100644 daemon/tsk.c create mode 100644 tests/tsk/Makefile.am create mode 100755 tests/tsk/test-fls0.sh create mode 100755 tests/tsk/test-icat.sh -- 2.7.0
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- appliance/packagelist.in | 3 ++ daemon/Makefile.am | 1 + daemon/tsk.c | 122 +++++++++++++++++++++++++++++++++++++++++++++++ generator/actions.ml | 33 +++++++++++++ po/POTFILES | 1 + src/MAX_PROC_NR | 2 +- 6 files changed, 161 insertions(+), 1 deletion(-) create mode 100644 daemon/tsk.c diff --git a/appliance/packagelist.in b/appliance/packagelist.in index aa5e07e..0951417 100644 --- a/appliance/packagelist.in +++ b/appliance/packagelist.in @@ -46,6 +46,7 @@ ifelse(REDHAT,1, pcre reiserfs-utils libselinux + sleuthkit syslinux-extlinux systemd dnl for /sbin/reboot and udevd vim-minimal @@ -89,6 +90,7 @@ dnl iproute has been renamed to iproute2 ntfs-3g openssh-client reiserfsprogs + sleuthkit sysvinit dnl for /sbin/reboot systemd dnl alternative for /sbin/reboot ufsutils @@ -120,6 +122,7 @@ ifelse(ARCHLINUX,1, ntfs-3g pcre reiserfsprogs + sleuthkit systemd vim xz diff --git a/daemon/Makefile.am b/daemon/Makefile.am index caba83e..4e2051b 100644 --- a/daemon/Makefile.am +++ b/daemon/Makefile.am @@ -179,6 +179,7 @@ guestfsd_SOURCES = \ syslinux.c \ tar.c \ truncate.c \ + tsk.c \ umask.c \ upload.c \ utimens.c \ diff --git a/daemon/tsk.c b/daemon/tsk.c new file mode 100644 index 0000000..8cc35eb --- /dev/null +++ b/daemon/tsk.c @@ -0,0 +1,122 @@ +/* libguestfs - the guestfsd daemon + * Copyright (C) 2009-2016 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <inttypes.h> +#include <string.h> +#include <unistd.h> + +#include "guestfs_protocol.h" +#include "daemon.h" +#include "actions.h" +#include "optgroups.h" + +static int file_out (const char *cmd); + +GUESTFSD_EXT_CMD(str_tsk_probe, icat); + +int +optgroup_icat_available (void) +{ + return prog_exists (str_tsk_probe); +} + +int +do_icat (const mountable_t *mountable, int64_t inode) +{ + CLEANUP_FREE char *cmd = NULL; + + /* Inode must be greater than 0 */ + if (inode < 0) { + reply_with_error ("inode must be >= 0"); + return -1; + } + + /* Construct the command. */ + if (asprintf (&cmd, "icat -r %s %" PRIi64, mountable->device, inode) == -1) { + reply_with_perror ("asprintf"); + return -1; + } + + return file_out (cmd); +} + +int +do_fls0 (const mountable_t *mountable) +{ + CLEANUP_FREE char *cmd = NULL; + + /* Construct the command. */ + if (asprintf (&cmd, "fls -pr %s", mountable->device) == -1) { + reply_with_perror ("asprintf"); + return -1; + } + + return file_out (cmd); +} + +static int +file_out (const char *cmd) +{ + int r; + FILE *fp; + char buffer[GUESTFS_MAX_CHUNK_SIZE]; + + if (verbose) + fprintf (stderr, "%s\n", cmd); + + fp = popen (cmd, "r"); + if (fp == NULL) { + reply_with_perror ("%s", cmd); + return -1; + } + + /* Now we must send the reply message, before the file contents. After + * this there is no opportunity in the protocol to send any error + * message back. Instead we can only cancel the transfer. + */ + reply (NULL, NULL); + + while ((r = fread (buffer, 1, sizeof buffer, fp)) > 0) { + if (send_file_write (buffer, r) < 0) { + pclose (fp); + return -1; + } + } + + if (ferror (fp)) { + fprintf (stderr, "fread: %m"); + send_file_end (1); /* Cancel. */ + pclose (fp); + return -1; + } + + if (pclose (fp) != 0) { + fprintf (stderr, "pclose: %m"); + send_file_end (1); /* Cancel. */ + return -1; + } + + if (send_file_end (0)) /* Normal end of file. */ + return -1; + + return 0; +} diff --git a/generator/actions.ml b/generator/actions.ml index 287d7f5..ff6aa3f 100644 --- a/generator/actions.ml +++ b/generator/actions.ml @@ -12919,6 +12919,39 @@ within the C<$Extend> folder. The filesystem from which to extract the file must be unmounted, otherwise the call will fail." }; + { defaults with + name = "icat"; added = (1, 33, 14); + style = RErr, [Mountable "device"; Int64 "inode"; FileOut "filename"], []; + proc_nr = Some 464; + optional = Some "icat"; + progress = true; cancellable = true; + shortdesc = "download a file to the local machine given its inode"; + longdesc = "\ +Download a file given its inode from the disk partition (eg. F</dev/sda1>) +and save it as F<filename> on the local machine. + +This allows to download deleted or inaccessible files." }; + + { defaults with + name = "fls0"; added = (1, 33, 14); + style = RErr, [Mountable "device"; FileOut "filename"], []; + proc_nr = Some 465; + optional = Some "icat"; + progress = true; cancellable = true; + shortdesc = "list the content of the disk"; + longdesc = "\ +This specialized command is used to get a listing of +the content of the disk partition (eg. F</dev/sda1>). +The list of filenames is written to the local file F<filenames> (on the host). + +The command can display hidden or deleted files. + +The output information contains the file type, the metadata address +and the full path of the file. + +If the file was deleted, an asterisk will be shown between the file type +and the metadata address." }; + ] (* Non-API meta-commands available only in guestfish. diff --git a/po/POTFILES b/po/POTFILES index 0fb99b0..4fbc551 100644 --- a/po/POTFILES +++ b/po/POTFILES @@ -108,6 +108,7 @@ daemon/sync.c daemon/syslinux.c daemon/tar.c daemon/truncate.c +daemon/tsk.c daemon/umask.c daemon/upload.c daemon/utimens.c diff --git a/src/MAX_PROC_NR b/src/MAX_PROC_NR index 26551cf..073c57b 100644 --- a/src/MAX_PROC_NR +++ b/src/MAX_PROC_NR @@ -1 +1 @@ -463 +465 -- 2.7.0
On Sun, Mar 06, 2016 at 05:42:25PM +0200, Matteo Cafasso wrote:
+static int +file_out (const char *cmd) +{ + int r; + FILE *fp; + char buffer[GUESTFS_MAX_CHUNK_SIZE];
Soon libguestfs will prevent you from using large stack allocations. This is easy to fix. See: https://www.redhat.com/archives/libguestfs/2016-March/msg00052.html
diff --git a/generator/actions.ml b/generator/actions.ml index 287d7f5..ff6aa3f 100644 --- a/generator/actions.ml +++ b/generator/actions.ml @@ -12919,6 +12919,39 @@ within the C<$Extend> folder. The filesystem from which to extract the file must be unmounted, otherwise the call will fail." };
+ { defaults with + name = "icat"; added = (1, 33, 14); + style = RErr, [Mountable "device"; Int64 "inode"; FileOut "filename"], []; + proc_nr = Some 464; + optional = Some "icat"; + progress = true; cancellable = true; + shortdesc = "download a file to the local machine given its inode"; + longdesc = "\ +Download a file given its inode from the disk partition (eg. F</dev/sda1>) +and save it as F<filename> on the local machine. + +This allows to download deleted or inaccessible files." };
This one looks OK.
+ { defaults with + name = "fls0"; added = (1, 33, 14); + style = RErr, [Mountable "device"; FileOut "filename"], []; + proc_nr = Some 465; + optional = Some "icat"; + progress = true; cancellable = true; + shortdesc = "list the content of the disk"; + longdesc = "\ +This specialized command is used to get a listing of +the content of the disk partition (eg. F</dev/sda1>). +The list of filenames is written to the local file F<filenames> (on the host). + +The command can display hidden or deleted files. + +The output information contains the file type, the metadata address +and the full path of the file. + +If the file was deleted, an asterisk will be shown between the file type +and the metadata address." };
As discussed, this shouldn't be a FileOut API at all. It should return a list of structs (RStructList). If the output could be very large then we may need to split it into an internal API, but let's not worry about that right now. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On 07/03/16 13:32, Richard W.M. Jones wrote:
On Sun, Mar 06, 2016 at 05:42:25PM +0200, Matteo Cafasso wrote:
+static int +file_out (const char *cmd) +{ + int r; + FILE *fp; + char buffer[GUESTFS_MAX_CHUNK_SIZE]; Soon libguestfs will prevent you from using large stack allocations. This is easy to fix. See: https://www.redhat.com/archives/libguestfs/2016-March/msg00052.html Will fix it.
diff --git a/generator/actions.ml b/generator/actions.ml index 287d7f5..ff6aa3f 100644 --- a/generator/actions.ml +++ b/generator/actions.ml @@ -12919,6 +12919,39 @@ within the C<$Extend> folder. The filesystem from which to extract the file must be unmounted, otherwise the call will fail." };
+ { defaults with + name = "icat"; added = (1, 33, 14); + style = RErr, [Mountable "device"; Int64 "inode"; FileOut "filename"], []; + proc_nr = Some 464; + optional = Some "icat"; + progress = true; cancellable = true; + shortdesc = "download a file to the local machine given its inode"; + longdesc = "\ +Download a file given its inode from the disk partition (eg. F</dev/sda1>) +and save it as F<filename> on the local machine. + +This allows to download deleted or inaccessible files." }; This one looks OK.
+ { defaults with + name = "fls0"; added = (1, 33, 14); + style = RErr, [Mountable "device"; FileOut "filename"], []; + proc_nr = Some 465; + optional = Some "icat"; + progress = true; cancellable = true; + shortdesc = "list the content of the disk"; + longdesc = "\ +This specialized command is used to get a listing of +the content of the disk partition (eg. F</dev/sda1>). +The list of filenames is written to the local file F<filenames> (on the host). + +The command can display hidden or deleted files. + +The output information contains the file type, the metadata address +and the full path of the file. + +If the file was deleted, an asterisk will be shown between the file type +and the metadata address." }; As discussed, this shouldn't be a FileOut API at all. It should return a list of structs (RStructList).
If the output could be very large then we may need to split it into an internal API, but let's not worry about that right now. This is exactly the internal API that's why I appended the 0 to the name xD.
As I explained in the previous e-mail. What is missing is the higher level API returning the RStructList.
Rich.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- Makefile.am | 1 + configure.ac | 1 + tests/tsk/Makefile.am | 27 +++++++++++++++++++++++++ tests/tsk/test-fls0.sh | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++ tests/tsk/test-icat.sh | 53 ++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 137 insertions(+) create mode 100644 tests/tsk/Makefile.am create mode 100755 tests/tsk/test-fls0.sh create mode 100755 tests/tsk/test-icat.sh diff --git a/Makefile.am b/Makefile.am index fe92630..42ff072 100644 --- a/Makefile.am +++ b/Makefile.am @@ -83,6 +83,7 @@ SUBDIRS += tests/journal SUBDIRS += tests/relative-paths SUBDIRS += tests/gdisk SUBDIRS += tests/regressions +SUBDIRS += tests/tsk endif # libguestfs-test-tool diff --git a/configure.ac b/configure.ac index fd8e534..c031707 100644 --- a/configure.ac +++ b/configure.ac @@ -281,6 +281,7 @@ AC_CONFIG_FILES([Makefile tests/selinux/Makefile tests/syslinux/Makefile tests/tmpdirs/Makefile + tests/tsk/Makefile tests/xfs/Makefile tests/xml/Makefile tools/Makefile diff --git a/tests/tsk/Makefile.am b/tests/tsk/Makefile.am new file mode 100644 index 0000000..b9a4195 --- /dev/null +++ b/tests/tsk/Makefile.am @@ -0,0 +1,27 @@ +# libguestfs +# Copyright (C) 2012-2016 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +include $(top_srcdir)/subdir-rules.mk + +TESTS = \ + test-icat.sh \ + test-fls0.sh + +TESTS_ENVIRONMENT = $(top_builddir)/run --test + +EXTRA_DIST = \ + $(TESTS) diff --git a/tests/tsk/test-fls0.sh b/tests/tsk/test-fls0.sh new file mode 100755 index 0000000..18c7690 --- /dev/null +++ b/tests/tsk/test-fls0.sh @@ -0,0 +1,55 @@ +#!/bin/bash - +# libguestfs +# Copyright (C) 2016 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Test the ntfscat-i command. + +set -e + +if [ -n "$SKIP_TEST_FLS0_SH" ]; then + echo "$0: test skipped because environment variable is set." + exit 77 +fi + +rm -f test-fls0.txt + +# Skip if TSK is not supported by the appliance. +if ! guestfish add /dev/null : run : available "icat"; then + echo "$0: skipped because TSK is not supported by the appliance" + exit 77 +fi + +if [ ! -s ../../test-data/phony-guests/ubuntu.img ]; then + echo "$0: skipped because ubuntu.img is zero-sized" + exit 77 +fi + +# list ubuntu image contents +guestfish --ro -a ../../test-data/phony-guests/ubuntu.img <<EOF +run +fls0 /dev/sda2 test-fls0.txt +EOF + +# test bin directory is in the list +grep -q "bin" test-fls0.txt + +if [ $? neq 0 ]; then + echo "$0: /bin not found in files list." + exit 1 +fi + +rm -f test-fls0.txt diff --git a/tests/tsk/test-icat.sh b/tests/tsk/test-icat.sh new file mode 100755 index 0000000..90356c4 --- /dev/null +++ b/tests/tsk/test-icat.sh @@ -0,0 +1,53 @@ +#!/bin/bash - +# libguestfs +# Copyright (C) 2016 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Test the ntfscat-i command. + +set -e + +if [ -n "$SKIP_TEST_ICAT_SH" ]; then + echo "$0: test skipped because environment variable is set." + exit 77 +fi + +rm -f test-mft.bin + +# Skip if TSK is not supported by the appliance. +if ! guestfish add /dev/null : run : available "icat"; then + echo "$0: skipped because TSK is not supported by the appliance" + exit 77 +fi + +if [ ! -s ../../test-data/phony-guests/windows.img ]; then + echo "$0: skipped because windows.img is zero-sized" + exit 77 +fi + +# download Master File Table ($MFT). +guestfish --ro -a ../../test-data/phony-guests/windows.img <<EOF +run +icat /dev/sda2 0 test-mft.bin +EOF + +# test extracted file is the Master File Table +if [ `head -c 5 test-mft.bin` != "FILE0" ]; then + echo "$0: wrong file extracted." + exit 1 +fi + +rm -f test-mft.bin -- 2.7.0
On Sun, Mar 06, 2016 at 05:42:24PM +0200, Matteo Cafasso wrote:
As discussed in the topic: https://www.redhat.com/archives/libguestfs/2016-March/msg00018.html
I'd like to add to libguestfs the disk forensics capabilities offered by The Sleuth Kit. http://www.sleuthkit.org/
The two APIs I'm adding with the patch are a simple example of which type of features TSK can enable.
A few comments in general terms: The current splitting of the commits doesn't make much sense to me. I think it would be better as: - commit to add TSK to the appliance - commit to add the icat API - tests for icat - commit to add the fls0 API - tests for fls0 although it would be fine to combine the tests with the new API, or even have all the tests as a single separate commit (as now). This benefits you because it will allow patches to go upstream earlier. For example, a commit to add TSK to the appliance is a simple and obvious change that I see no problem with. Also the icat API is closer to being ready than the fls0 API (see below for explanation).
<fs> fls0 /dev/sda2 /home/noxdafox/disk-content.txt
r/r 15711-128-1: $Recycle.Bin/S-1-5-21-2379395878-2832339042-1309242031-1000/desktop.ini -/r * 60015-128-1: $Recycle.Bin/S-1-5-21-2379395878-2832339042-1309242031-1000/$R07QQZ2.txt -/r * 60015-128-3: $Recycle.Bin/S-1-5-21-2379395878-2832339042-1309242031-1000/$R07QQZ2.txt:Zone.Identifier
What is `/home/noxdafox/disk-content.txt'? The problem with this API is it pushes all the parsing up in the stack, to libguestfs consumers. In general we'd like to avoid that and have just one place where all parsing needs to be done (ie. libguestfs itself), so it'd be nicer to have an API that returns a list of structs (RStructList) with all the important fields parsed out. Does TSK have a machine-readable mode? If it does, it'll definitely make things easier if (eg) JSON or XML output is available. If not, push upstream to add that to TSK -- it's a simple change for them, which will make their tools much more usable, a win for everyone. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Sun, Mar 06, 2016 at 05:42:24PM +0200, Matteo Cafasso wrote:
As discussed in the topic: https://www.redhat.com/archives/libguestfs/2016-March/msg00018.html
I'd like to add to libguestfs the disk forensics capabilities offered by The Sleuth Kit. http://www.sleuthkit.org/
The two APIs I'm adding with the patch are a simple example of which type of features TSK can enable. A few comments in general terms:
The current splitting of the commits doesn't make much sense to me. I think it would be better as:
- commit to add TSK to the appliance
- commit to add the icat API
- tests for icat
- commit to add the fls0 API
- tests for fls0
although it would be fine to combine the tests with the new API, or even have all the tests as a single separate commit (as now).
This benefits you because it will allow patches to go upstream earlier. For example, a commit to add TSK to the appliance is a simple and obvious change that I see no problem with. Also the icat API is closer to being ready than the fls0 API (see below for explanation). Indeed I've done quite a poor job in this. I will split it as suggested.
<fs> fls0 /dev/sda2 /home/noxdafox/disk-content.txt r/r 15711-128-1: $Recycle.Bin/S-1-5-21-2379395878-2832339042-1309242031-1000/desktop.ini -/r * 60015-128-1: $Recycle.Bin/S-1-5-21-2379395878-2832339042-1309242031-1000/$R07QQZ2.txt -/r * 60015-128-3: $Recycle.Bin/S-1-5-21-2379395878-2832339042-1309242031-1000/$R07QQZ2.txt:Zone.Identifier What is `/home/noxdafox/disk-content.txt'? It's the local (host side) file where to store the command output.
The problem with this API is it pushes all the parsing up in the stack, to libguestfs consumers.
In general we'd like to avoid that and have just one place where all parsing needs to be done (ie. libguestfs itself), so it'd be nicer to have an API that returns a list of structs (RStructList) with all the important fields parsed out. As the API documentation says, this is the low level API which I have
On 07/03/16 13:29, Richard W.M. Jones wrote: provided as an example. I took inspiration from the guestfs_ls0 API which does a similar job storing the content of a directory onto a host file. If I understood correctly (the dynamic code generation is still confusing me a bit), the way Libguestfs implements commands which could have a large output is via first dumping it onto a local file and then iterating over it. This command would list the entire content of a disk including the deleted files therefore we need to expect a large output. What is missing is the higher level implementation which would pretty much look like the libguestfs_ls API. I need to better understand how to implement it and suggestions are more than appreciated. I tried to trace back how the guestfs_find is implemented for example, but I'm still a bit disoriented by the automagic code generation.
Does TSK have a machine-readable mode? If it does, it'll definitely make things easier if (eg) JSON or XML output is available. If not, push upstream to add that to TSK -- it's a simple change for them, which will make their tools much more usable, a win for everyone.
I personally disagree on this. The TSK `fls` command is a clone of the bash `ls` one. Maybe it's more similar to `ls -al` as it returns additional information. IMHO asking to upstream to add JSON or XML output format would sound pretty much as asking the same to bash for the `ls` utility. The end result is to still return a list of structs or a list of strings. But parsing the `fls` output shouldn't be that hard. It's documentation is here: http://wiki.sleuthkit.org/index.php?title=Fls
Rich.
On Mon, Mar 07, 2016 at 08:14:41PM +0200, noxdafox wrote:
As the API documentation says, this is the low level API which I have provided as an example.
I took inspiration from the guestfs_ls0 API which does a similar job storing the content of a directory onto a host file.
If I understood correctly (the dynamic code generation is still confusing me a bit), the way Libguestfs implements commands which could have a large output is via first dumping it onto a local file and then iterating over it. This command would list the entire content of a disk including the deleted files therefore we need to expect a large output.
Your understanding is correct. But the fls0 API still isn't safe because (I assume) it cannot handle filenames containing '\n'. There's another API which handles arbitrary length RStructList returns, which is: guestfs_lxattrlist / guestfs_internal_lxattrlist (see src/file.c:guestfs_impl_lxattrlist and daemon/xattr.c).
What is missing is the higher level implementation which would pretty much look like the libguestfs_ls API. I need to better understand how to implement it and suggestions are more than appreciated. I tried to trace back how the guestfs_find is implemented for example, but I'm still a bit disoriented by the automagic code generation.
See guestfs_impl_ls in src/file.c. All non_daemon_functions are implemented by some guestfs_impl_* function in the library side.
Does TSK have a machine-readable mode? If it does, it'll definitely make things easier if (eg) JSON or XML output is available. If not, push upstream to add that to TSK -- it's a simple change for them, which will make their tools much more usable, a win for everyone. I personally disagree on this. The TSK `fls` command is a clone of the bash `ls` one. Maybe it's more similar to `ls -al` as it returns additional information. IMHO asking to upstream to add JSON or XML output format would sound pretty much as asking the same to bash for the `ls` utility.
The end result is to still return a list of structs or a list of strings. But parsing the `fls` output shouldn't be that hard. It's documentation is here: http://wiki.sleuthkit.org/index.php?title=Fls
Well I still think it would be better to make this parsable. If I want to get information about a file in a shell script, I use the 'stat(1)' program since that has machine-readable output (stat -c). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
On Mon, Mar 07, 2016 at 08:14:41PM +0200, noxdafox wrote:
As the API documentation says, this is the low level API which I have provided as an example.
I took inspiration from the guestfs_ls0 API which does a similar job storing the content of a directory onto a host file.
If I understood correctly (the dynamic code generation is still confusing me a bit), the way Libguestfs implements commands which could have a large output is via first dumping it onto a local file and then iterating over it. This command would list the entire content of a disk including the deleted files therefore we need to expect a large output. Your understanding is correct. But the fls0 API still isn't safe because (I assume) it cannot handle filenames containing '\n'. I haven't considered this issue. This is why guestfs_ls0 separates the results using a '\0' character right? I'll try to reproduce this and see how TSK gives me the output.
There's another API which handles arbitrary length RStructList returns, which is: guestfs_lxattrlist / guestfs_internal_lxattrlist (see src/file.c:guestfs_impl_lxattrlist and daemon/xattr.c). I will take a look at these ones thanks!
What is missing is the higher level implementation which would pretty much look like the libguestfs_ls API. I need to better understand how to implement it and suggestions are more than appreciated. I tried to trace back how the guestfs_find is implemented for example, but I'm still a bit disoriented by the automagic code generation. See guestfs_impl_ls in src/file.c. All non_daemon_functions are implemented by some guestfs_impl_* function in the library side. I guess I'll come back with a complete solution with both low level and high level implementation.
Does TSK have a machine-readable mode? If it does, it'll definitely make things easier if (eg) JSON or XML output is available. If not, push upstream to add that to TSK -- it's a simple change for them, which will make their tools much more usable, a win for everyone. I personally disagree on this. The TSK `fls` command is a clone of the bash `ls` one. Maybe it's more similar to `ls -al` as it returns additional information. IMHO asking to upstream to add JSON or XML output format would sound pretty much as asking the same to bash for the `ls` utility.
The end result is to still return a list of structs or a list of strings. But parsing the `fls` output shouldn't be that hard. It's documentation is here: http://wiki.sleuthkit.org/index.php?title=Fls Well I still think it would be better to make this parsable. If I want to get information about a file in a shell script, I use the 'stat(1)' program since that has machine-readable output (stat -c). Indeed but in such case you know what to expect as the set of information is a closed and well defined one. In this case the options are unfortunately many. I can of course propose
On 07/03/16 21:31, Richard W.M. Jones wrote: the idea to upstream but I guess they won't like it much.
Rich.
participants (3)
-
Matteo Cafasso -
noxdafox
-
Richard W.M. Jones