hivexml - Flattened vs. Expanded XML
All, Greetings. I am new to this mailing list. We have been working with XML for digital forensics. One of the areas that we wish to create a schema for is the representation of registry entries. We are interested in hivexml as a tool for extracting the registry as an XML representation. In our discussion with possible users, we have generally come to the conclusion that it is useful to represent each XML key as a fully expanded path, rather than preserving the tree structure of the registry hive. Although this may seem verbose, it makes processing the data significantly easier. Is working with the hivexml system in a production environment? If so, do you have any thoughts on this matter? You can find an example of the digital forensics XML at: http://www.forensicswiki.org/wiki/Fiwalk Regards, Simson Garfinkel
This would be very much dependent on the kind of processing desired; I can immediately see several XPath queries I might want to write which would be unwieldy to represent without the tree structure preserved. Flattening the document removes much of the utility of XML-based toolchains, while still paying a penalty in storage size and parser complexity; at that point, why not just export to the conventional .reg text format? On Fri, Mar 19, 2010 at 3:45 PM, Simson Garfinkel <simsong@acm.org> wrote:
All,
Greetings. I am new to this mailing list.
We have been working with XML for digital forensics. One of the areas that we wish to create a schema for is the representation of registry entries.
We are interested in hivexml as a tool for extracting the registry as an XML representation.
In our discussion with possible users, we have generally come to the conclusion that it is useful to represent each XML key as a fully expanded path, rather than preserving the tree structure of the registry hive. Although this may seem verbose, it makes processing the data significantly easier.
Is working with the hivexml system in a production environment? If so, do you have any thoughts on this matter?
You can find an example of the digital forensics XML at: http://www.forensicswiki.org/wiki/Fiwalk
Regards,
Simson Garfinkel
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
We are interested in seeing if specific registry keys are present or not. The problem with the standard .reg text format is that we would then need to write a parser for it, rather than simply using our XML parser. However, we haven't made a decision on this. That's why I sent the query. Yes, I can imagine several XPath queries that I might want to write as well. But I don't know if anybody is actually doing that. Hence my question. Is anyone using XML for registry entries in a production environment? On Mar 19, 2010, at 2:34 PM, Charles Duffy wrote:
This would be very much dependent on the kind of processing desired; I can immediately see several XPath queries I might want to write which would be unwieldy to represent without the tree structure preserved.
Flattening the document removes much of the utility of XML-based toolchains, while still paying a penalty in storage size and parser complexity; at that point, why not just export to the conventional .reg text format?
On Fri, Mar 19, 2010 at 3:45 PM, Simson Garfinkel <simsong@acm.org> wrote: All,
Greetings. I am new to this mailing list.
We have been working with XML for digital forensics. One of the areas that we wish to create a schema for is the representation of registry entries.
We are interested in hivexml as a tool for extracting the registry as an XML representation.
In our discussion with possible users, we have generally come to the conclusion that it is useful to represent each XML key as a fully expanded path, rather than preserving the tree structure of the registry hive. Although this may seem verbose, it makes processing the data significantly easier.
Is working with the hivexml system in a production environment? If so, do you have any thoughts on this matter?
You can find an example of the digital forensics XML at: http://www.forensicswiki.org/wiki/Fiwalk
Regards,
Simson Garfinkel
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
On Fri, Mar 19, 2010 at 01:45:34PM -0700, Simson Garfinkel wrote:
Greetings. I am new to this mailing list.
We have been working with XML for digital forensics. One of the areas that we wish to create a schema for is the representation of registry entries.
We are interested in hivexml as a tool for extracting the registry as an XML representation.
'hivexml' is really just a demo program that I wrote. You can change the XML format or even rewrite it -- the whole program is only 345 lines of code. http://git.annexia.org/?p=hivex.git;a=blob;f=xml/hivexml.c;hb=HEAD If you are interested in forensic analysis, it might be worth looking at the analysis tools we wrote as well / instead: http://git.annexia.org/?p=hivex.git;a=tree;f=lib/tools;hb=HEAD These analysis tools look at the registry in much more detail and can look for inconsistencies, hidden keys etc. which we don't care so much about in the main hivex library. One issue that may be of concern is string encoding in registry values, which is not well defined. Naturally for XML I suppose you'd want to represent string values as UTF-8. However it's almost impossible to know for sure how strings are encoded in the registry, so doing this conversion would either involve a heuristic, or you'd have to store binary blobs in the XML (encoded as Base64 or as hex digits). The registry is a mess in this respect. [...]
You can find an example of the digital forensics XML at: http://www.forensicswiki.org/wiki/Fiwalk
Looks interesting. It should be easily possible to get libguestfs to write this format for disk images. There is already a (trivial) demo program I wrote along those lines: http://git.annexia.org/?p=libguestfs.git;a=blob;f=examples/to-xml.c;hb=HEAD - - - If you have changes for libguestfs or hivex, please submit them to this mailing list as for any open source project: http://people.redhat.com/~rjones/how-to-supply-code-to-open-source-projects/ Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
One issue that may be of concern is string encoding in registry values, which is not well defined. Naturally for XML I suppose you'd want to represent string values as UTF-8. However it's almost impossible to know for sure how strings are encoded in the registry, so doing this conversion would either involve a heuristic, or you'd have to store binary blobs in the XML (encoded as Base64 or as hex digits). The registry is a mess in this respect.
Rich, We have encountered the same problem with the XML encoding of file names. Sometimes they are in ASCII, sometimes they are in a code page, sometimes they are in UTF-8, and sometimes they are in corrupt UTF-8. This is the approach we are using: 1. Represent everything that can be represented in UTF-8 as UTF-8. 2. If something can't be shown as UTF-8, then we add a "coding='base64'" attribute the XML tag and represent it as Base64 We would like to replace #2 with an explicit encoding of the invalid characters as Unicode entities, but we haven't written that.
[...]
You can find an example of the digital forensics XML at: http://www.forensicswiki.org/wiki/Fiwalk
Looks interesting. It should be easily possible to get libguestfs to write this format for disk images. There is already a (trivial) demo program I wrote along those lines:
http://git.annexia.org/?p=libguestfs.git;a=blob;f=examples/to-xml.c;hb=HEAD
Thanks. I'll check that out. We've made a lot of progress writing program in Python that process the Digital Forensics XML, and it is proving to be a good approach for integrating a range of computer forensic tools. You may be interested in my paper: Garfinkel, Simson., Automating Disk Forensic Processing with SleuthKit, XML and Python, Systematic Approaches to Digital Forensics Engineering (IEEE/SADFE 2009), Oakland, California. http://simson.net/clips/academic/2009.SADFE.xml_forensics.pdf
- - -
If you have changes for libguestfs or hivex, please submit them to this mailing list as for any open source project:
http://people.redhat.com/~rjones/how-to-supply-code-to-open-source-projects/
Thanks. My understanding is that the current code does not build on MacOS. I was just going to download the GIT repository and have at it, but I was not sure how to send back changes.
On Sat, Mar 20, 2010 at 09:18:58AM -0700, Simson Garfinkel wrote:
Thanks. My understanding is that the current code does not build on MacOS.
No, it won't build on Mac OS X even slightly .. We provide builds for Fedora, Debian and Ubuntu so far: http://libguestfs.org/FAQ.html#binaries and you'll probably want to read this too: http://libguestfs.org/FAQ.html#distros (Best bet for Mac OS X is probably option #3).
I was just going to download the GIT repository and have at it, but I was not sure how to send back changes.
Commit the changes to your local git repository and use git format-patch to format them into something which can be posted on the mailing list. There's tutorials out there which cover this I'm sure. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://et.redhat.com/~rjones/libguestfs/ See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html
participants (3)
-
Charles Duffy -
Richard W.M. Jones -
Simson Garfinkel