[PATCH v2 0/2] Added download_blocks API
v2: - Rebase on top of master Matteo Cafasso (2): New API: download_blocks Added download_blocks API test daemon/sleuthkit.c | 41 ++++++++++++++++++++++++++- generator/actions.ml | 24 ++++++++++++++++ gobject/Makefile.inc | 2 ++ src/MAX_PROC_NR | 2 +- tests/tsk/Makefile.am | 1 + tests/tsk/test-download-blocks.sh | 58 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 126 insertions(+), 2 deletions(-) create mode 100755 tests/tsk/test-download-blocks.sh -- 2.8.1
This function allows to download file system data units (blocks) from the given partition. The API can be used to detect data hidden within filesystem bad blocks or slack space. Moreover for filesystems such as Ext3 and Ext4, this function is the only way to retrieve deleted files. An example is given in the function tests. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- daemon/sleuthkit.c | 41 ++++++++++++++++++++++++++++++++++++++++- generator/actions.ml | 24 ++++++++++++++++++++++++ gobject/Makefile.inc | 2 ++ src/MAX_PROC_NR | 2 +- 4 files changed, 67 insertions(+), 2 deletions(-) diff --git a/daemon/sleuthkit.c b/daemon/sleuthkit.c index 1262b68..eb7f877 100644 --- a/daemon/sleuthkit.c +++ b/daemon/sleuthkit.c @@ -32,6 +32,7 @@ static int send_command_output (const char *cmd); GUESTFSD_EXT_CMD(str_icat, icat); +GUESTFSD_EXT_CMD(str_blkls, blkls); int do_download_inode (const mountable_t *mountable, int64_t inode) @@ -46,7 +47,8 @@ do_download_inode (const mountable_t *mountable, int64_t inode) } /* Construct the command. */ - ret = asprintf(&cmd, "%s -r %s %" PRIi64, str_icat, mountable->device, inode); + ret = asprintf (&cmd, "%s -r %s %" PRIi64, + str_icat, mountable->device, inode); if (ret < 0) { reply_with_perror ("asprintf"); return -1; @@ -55,6 +57,43 @@ do_download_inode (const mountable_t *mountable, int64_t inode) return send_command_output (cmd); } +/* Takes optional arguments, consult optargs_bitmask. */ +int +do_download_blocks (const mountable_t *mountable, int64_t start, int64_t stop, + int unallocated) +{ + int ret; + const char *params; + CLEANUP_FREE char *cmd = NULL; + + /* Data unit address start must be greater than 0 */ + if (start < 0) { + reply_with_error ("starting address must be greater than zero"); + return -1; + } + + /* Data unit address end must be greater than start */ + if (stop <= start) { + reply_with_error ("stopping address must greater than starting address"); + return -1; + } + + if (!(optargs_bitmask & GUESTFS_DOWNLOAD_BLOCKS_UNALLOCATED_BITMASK)) + params = " -e"; + else + params = ""; + + /* Construct the command. */ + ret = asprintf (&cmd, "%s %s %s %" PRIi64 "-%" PRIi64, + str_blkls, mountable->device, params, start, stop); + if (ret < -0) { + reply_with_perror ("asprintf"); + return -1; + } + + return send_command_output (cmd); +} + /* Run the given command, collect the output and send it to the appliance. * Return 0 on success, -1 on error. */ diff --git a/generator/actions.ml b/generator/actions.ml index 4423313..10f291f 100644 --- a/generator/actions.ml +++ b/generator/actions.ml @@ -13177,6 +13177,30 @@ The optional C<force> boolean controls whether the context is reset for customizable files, and also whether the user, role and range parts of the file context is changed." }; + { defaults with + name = "download_blocks"; added = (1, 33, 44); + style = RErr, [Mountable "device"; Int64 "start"; Int64 "stop"; FileOut "filename"], [OBool "unallocated"]; + proc_nr = Some 468; + optional = Some "sleuthkit"; + progress = true; cancellable = true; + shortdesc = "download the given data units from the disk"; + longdesc = "\ +Download the data units from F<start> address +to F<stop> from the disk partition (eg. F</dev/sda1>) +and save them as F<filename> on the local machine. + +The use of this API on sparse disk image formats such as QCOW, +may result in large zero-filled files downloaded on the host. + +The size of a data unit varies across filesystem implementations. +On NTFS filesystems data units are referred as clusters +while on ExtX ones they are referred as fragments. + +If the optional C<unallocated> flag is true (default is false), +only the unallocated blocks will be extracted. +This is useful to detect hidden data or to retrieve deleted files +which data units have not been overwritten yet." }; + ] (* Non-API meta-commands available only in guestfish. diff --git a/gobject/Makefile.inc b/gobject/Makefile.inc index 797bb33..7698c2a 100644 --- a/gobject/Makefile.inc +++ b/gobject/Makefile.inc @@ -68,6 +68,7 @@ guestfs_gobject_headers= \ include/guestfs-gobject/optargs-copy_file_to_file.h \ include/guestfs-gobject/optargs-cpio_out.h \ include/guestfs-gobject/optargs-disk_create.h \ + include/guestfs-gobject/optargs-download_blocks.h \ include/guestfs-gobject/optargs-e2fsck.h \ include/guestfs-gobject/optargs-fstrim.h \ include/guestfs-gobject/optargs-glob_expand.h \ @@ -157,6 +158,7 @@ guestfs_gobject_sources= \ src/optargs-copy_file_to_file.c \ src/optargs-cpio_out.c \ src/optargs-disk_create.c \ + src/optargs-download_blocks.c \ src/optargs-e2fsck.c \ src/optargs-fstrim.c \ src/optargs-glob_expand.c \ diff --git a/src/MAX_PROC_NR b/src/MAX_PROC_NR index 5873851..1023289 100644 --- a/src/MAX_PROC_NR +++ b/src/MAX_PROC_NR @@ -1 +1 @@ -467 +468 -- 2.8.1
The test shows how the function can be used to retrieve deleted data from a disk partition. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- tests/tsk/Makefile.am | 1 + tests/tsk/test-download-blocks.sh | 58 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) create mode 100755 tests/tsk/test-download-blocks.sh diff --git a/tests/tsk/Makefile.am b/tests/tsk/Makefile.am index f9b2fef..0b50839 100644 --- a/tests/tsk/Makefile.am +++ b/tests/tsk/Makefile.am @@ -19,6 +19,7 @@ include $(top_srcdir)/subdir-rules.mk TESTS = \ test-download-inode.sh \ + test-download-blocks.sh \ test-filesystem-walk.sh TESTS_ENVIRONMENT = $(top_builddir)/run --test diff --git a/tests/tsk/test-download-blocks.sh b/tests/tsk/test-download-blocks.sh new file mode 100755 index 0000000..234b557 --- /dev/null +++ b/tests/tsk/test-download-blocks.sh @@ -0,0 +1,58 @@ +#!/bin/bash - +# libguestfs +# Copyright (C) 2016 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Test the download_blocks command. + +set -e + +if [ -n "$SKIP_TEST_DOWNLOAD_BLOCKS_SH" ]; then + echo "$0: test skipped because environment variable is set." + exit 77 +fi + +rm -f test-download-blocks.bin + +# Skip if TSK is not supported by the appliance. +if ! guestfish add /dev/null : run : available "sleuthkit"; then + echo "$0: skipped because TSK is not available in the appliance" + exit 77 +fi + +if [ ! -s ../../test-data/phony-guests/blank-fs.img ]; then + echo "$0: skipped because blank-fs.img is zero-sized" + exit 77 +fi + +# download Master File Table ($MFT). +guestfish --ro -a ../../test-data/phony-guests/blank-fs.img <<EOF +run +mount /dev/sda1 / +write /test.txt "$foo$bar$" +rm /test.txt +umount / +download-blocks /dev/sda1 0 8192 test-download-blocks.bin unallocated:true +EOF + +# test extracted data contains $foo$bar$ string +grep -q "$foo$bar$" test-download-blocks.bin +if [ $? neq 0 ]; then + echo "$0: removed data not found." + exit 1 +fi + +rm -f test-download-blocks.bin -- 2.8.1
Hi, On Sunday, 17 July 2016 20:40:18 CEST Matteo Cafasso wrote:
v2:
- Rebase on top of master
Matteo Cafasso (2): New API: download_blocks Added download_blocks API test
daemon/sleuthkit.c | 41 ++++++++++++++++++++++++++- generator/actions.ml | 24 ++++++++++++++++ gobject/Makefile.inc | 2 ++ src/MAX_PROC_NR | 2 +- tests/tsk/Makefile.am | 1 + tests/tsk/test-download-blocks.sh | 58 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 126 insertions(+), 2 deletions(-) create mode 100755 tests/tsk/test-download-blocks.sh
The series LGTM, I pushed it after removing the extra change in do_download_inode and fixing the version. More a curiosity question than a complain or something else: how are these APIs are supposed to be used? What is the forensics-related workflow using them? Considering they are quite specific, I was thinking about adding a documentation paragraph and/or some example to describe/show them better, what do you think? Thanks, -- Pino Toscano
On 20/07/16 17:04, Pino Toscano wrote:
Hi,
On Sunday, 17 July 2016 20:40:18 CEST Matteo Cafasso wrote:
v2:
- Rebase on top of master
Matteo Cafasso (2): New API: download_blocks Added download_blocks API test
daemon/sleuthkit.c | 41 ++++++++++++++++++++++++++- generator/actions.ml | 24 ++++++++++++++++ gobject/Makefile.inc | 2 ++ src/MAX_PROC_NR | 2 +- tests/tsk/Makefile.am | 1 + tests/tsk/test-download-blocks.sh | 58 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 126 insertions(+), 2 deletions(-) create mode 100755 tests/tsk/test-download-blocks.sh The series LGTM, I pushed it after removing the extra change in do_download_inode and fixing the version.
More a curiosity question than a complain or something else: how are these APIs are supposed to be used? What is the forensics-related workflow using them? Current focus is deleted/unaccessible files retrieval as I believe this is the most interesting feature for libguestfs users.
A forensic workflow example would be: * start libguestfs and identify the disk partition where your data is * run filesystem_walk to get list of files which are visible within that disk partition * if the deleted file you want to recover is in that list, you'll get its inode * use download_inode to try recovering the deleted file For Ext3+ filesystems the thing is a bit more complicated. These filesystems remove the block links when the file gets deleted making its recovery more difficult. Only choice is carving out the data and download_blocks is the function which allows you to do so. What the User needs is an API capable of mapping disk blocks to files and then he/she will be able to recover them using download_blocks. Most of the APIs I am introducing are inspired from TSK ones. Here's a more detailed example on how to retrieve deleted data from disks. http://wiki.sleuthkit.org/index.php?title=FS_Analysis Afterwards, we could focus on more interesting topics such as evidence gathering and forensics analysis. Automating it is a challenging topic as most of the "evidence reconstruction" requires careful thinking as the data might have been tampered or obfuscated. Yet there are quite interesting features we can add which could support forensic analysis as well as cloud security solutions. Think about libguestfs scanning Open Stack instance disks to detect anomalies within cloud deployments. libguestfs is the perfect tool as it easily allows to abstract both the disk virtualisation technology (qcow2, vmdk etc..) and the guest Operating System. You can find an example on libguestfs-based VM scanning solution in here: https://github.com/noxdafox/vminspect If you check the "timeline" command implementation, you'll find few of the new APIs in use.
Considering they are quite specific, I was thinking about adding a documentation paragraph and/or some example to describe/show them better, what do you think?
This is a good question, I was thinking about a blog post to start with but a paragraph in the documentation sounds good as well. Let me know if you need help for that, I can provide some real-life example.
Thanks,
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
participants (3)
-
Matteo Cafasso -
noxdafox
-
Pino Toscano