Is there a reason why a previous version of hivex ignored this condition while more recent ones throw an error? I do have some hive files that are failing (I'm yet to validate if they are good or were corrupted some other way). I will do so and share my findings Thanks ~ Hari On 9/24/13 5:20 PM, "Richard W.M. Jones" <rjones@redhat.com> wrote:

On Tue, Sep 24, 2013 at 10:14:12PM +0100, Richard W.M. Jones wrote:

...

[Forwarding to the proper list]

We use hivex for one of our projects and I see the following diff committed to the hivex project (source link provided) that ignores trailing garbage. But this code has been reverted to no ignore this condition in later version of hivex

- /* This error is seemingly common in uncorrupt registry files. */ - /* - fprintf (stderr, "hivex: %s: ignoring trailing garbage at end of file (at %zu, after %zu pages)\n", - filename, off, h->pages); - */ + /* NB: This error is seemingly common in uncorrupt registry files. */ + if (h->msglvl >= 2) + fprintf (stderr, "hivex: %s: ignoring trailing garbage at end of file (at %zu, after %zu pages)\n", + filename, off, h->pages); Break;

http://git.annexia.org/?p=hivex.git;a=commitdiff;h=5439842dc591accbf620a5 f76dd6f04ad4d5c971

...

In your experience, is this a common condition seen in some windows hive files and is it safe to ignore when this condition is encountered?

I have seen this, rarely, in registry hives. I don't understand at all why it would happen, but would welcome known-good hives that demonstrate the problem. Remember that the registry format is completely undocumented, so it's all guesswork :-(

The commit you mention just changes the message so it is printed when debugging is enabled, but is otherwise the same, so I'm not sure what precisely the problem is. It should just print the message (if debugging is enabled) and continue.

Rich.

-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

Hi Rich, Sorry about the delayed response. I understand that since hive formats are not open, the current tack of warning about the hive file is a better one To respond to you question about "whether it fails", hivexsh is unable to open the hive file and it prints this message and exits. I've attached the verbose logs as requested This extract from the logs shows that the hivexsh complains content after file offset 0x77c000 is garbage hivex: badsys: trailing garbage at end of file (at 0x77c000, after 1849 pages) So, I went ahead and truncated the contents of the file after that file offset and hivex was able to successfully open the new hive file The sizes of the different hive files shown here [root@ip-10-66-209-246 ~]# ls -als bads* 7664 -rw--w--w- 1 root root 7847936 Oct 1 21:51 badsys (SYSTEM hive file after truncating trailing garbage) 7680 -rw------- 1 root root 7864320 Oct 1 21:21 badsysback (original SYSTEM hive file) 16 -rw-r--r-- 1 root root 16384 Oct 1 21:55 badsysgarb (16K of truncated garbage) Looking into hive file, the last 16K is what hivex complains to be trailing garbage. All of that 16K is zeroes. [root@ip-10-66-209-246 ~]# dd if=./badsysback of=./badsysgarb bs=4k skip=1916 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 9.1288e-05 s, 179 MB/s [root@ip-10-66-209-246 ~]# hexdump badsysgarb 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0004000 I'm not sure what the filesystem cluster size is but could the SYSTEM hive file be padded with 16K of zeroes to match 32k cluster allocation (or something like that?) I imported both hive files (original one that hivex complained about and the truncated one) into a windows regedit and the tool opened them both fine and showed pretty much the same content I'm going to see if the hive file I have has any sensitive customer information and will try to share it if it does not Thanks for you help! ~ Hari On 9/25/13 4:59 AM, "Richard W.M. Jones" <rjones@redhat.com> wrote:

On Tue, Sep 24, 2013 at 09:31:26PM +0000, Subramanian, Hari wrote:

...

Is there a reason why a previous version of hivex ignored this condition while more recent ones throw an error?

We don't understand all aspects of the hive format, so it's better to warn about these things than ignore them.

...

I do have some hive files that are failing (I'm yet to validate if they are good or were corrupted some other way). I will do so and share my findings

When you say "that are failing" do you mean they actually fail (and if so how -- full verbose logs please) or they print this warning, which is not a failure?

Rich.

-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

On Fri, Oct 04, 2013 at 02:12:08PM +0000, Subramanian, Hari wrote:

To respond to you question about "whether it fails", hivexsh is unable to open the hive file and it prints this message and exits. I've attached the verbose logs as requested

OK, I understand it now. It is in fact failing, setting errno = ENOSYS and returning an error.

This extract from the logs shows that the hivexsh complains content after file offset 0x77c000 is garbage

hivex: badsys: trailing garbage at end of file (at 0x77c000, after 1849 pages)

So, I went ahead and truncated the contents of the file after that file offset and hivex was able to successfully open the new hive file

I guess if the hive comes from a real guest we can change this to warn but not fail. Should be a pretty simple patch. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/

On Fri, Oct 04, 2013 at 03:13:47PM +0000, Subramanian, Hari wrote:

Rich,

Our workflow is something like this:

0. Start with a fresh copy of windows server 2k8 1. We read the system hive and then write to it a bunch of times 2. Boot windows 3. Read from the system hive

Hivex reports the failure at step #3. I also noticed that the size of the registry hive observed in step #3 is the same as step #0. Is it possible that hivex issues write that cause a hive file to shrink in size and while compacting the hive file, it retains the size but zeroes out the end of the file?

Ah so hivex wrote the hive? That could indicate a bug in hivex or perhaps some synchronization issue when uploading it into the guest? Anyway hivex itself never shrinks the hive. In fact it only ever appends to the end of the hive, extending it if necessary (it's not always necessary). When it extends the hive it's supposed to be initializing the hbin correctly. On the other hand it's anyone's guess what Windows does to the hive in step 2. BTW the source is all open ... https://github.com/libguestfs/hivex/blob/master/lib/write.c

That would point to the trailing zeroes getting introduced in step #1

It's also possible that windows is padding those zeroes in step #2. I'm adding some instrumentation to confirm this

A good start would be logging the size and MD5 checksum of the hive before and after each step, and also from within Windows (IIRC there is an MD5 checksum program in PowerShell). That would rule out some sort of non-hivex-related corruption when you're downloading or uploading the hive. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

On Tue, Oct 08, 2013 at 11:16:05PM +0000, Subramanian, Hari wrote:

Hi Rich,

I'm still working on validating whether the trailing zeroes were introduced by hivex or by windows (though I highly doubt it's hivex). But since it's part of a more complex workflow which is not that easy to modify, it's still a work in progress

In the meanwhile, I wanted to be sure that ignoring the condition when hivex saw these trailing zeroes was the right solution. I applied the following patch to hivex. Not sure if this is what you had in mind

[ec2-user@ip-10-66-218-126 hivex-1.3.8]$ diff handle.c handle-mod.c 222,226c222,223 < SET_ERRNO (ENOTSUP, < "%s: trailing garbage at end of file " < "(at 0x%zx, after %zu pages)", < filename, off, pages); < goto error; ---

...

printf("hivex: %s: trailing garbage at end of file (at 0x%zx, after %zu pages)\n", filename, off, pages); break;

I wanted to understand the behavior of hivex when we wrote to the end of the hive file (added a new entry under ControlSet001\services). When I do that I get the following error

badsys-1-win-add-reg\ControlSet001\services> ls hivex: _hivex_get_children: returning EFAULT because: subkey_lf is not a valid block (0x780020) ls: Bad address

This is not a good sign at all. It indicates a corrupt hive because a pointer in the hive points (probably) beyond the end of the hive. I would guess the hive has been truncated.

Obviously the add/modify was not successful. But regedit parsed the hive successfully.

Depending on how you used Windows regedit, it might not visit the corrupted part of the hive, or it might just ignore the corruption. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top