[PATCH 0/13 v2] Prepare for adding write support to hivex (Windows registry) library
This series of patches, now up to 13 parts, contains lots of bug fixes and groundwork required before we add write support to the hivex library. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
Richard W.M. Jones wrote:
Subject: [PATCH 03/13] hivex: Collect more statistics about registries.
This looks fine. Mostly mechanical, like s/h->var/var/, ...
- filename, off, h->pages); + filename, off, pages);
... with type adjustments
- int32_t seg_len; + size_t seg_len;
However, I find the following hard to read: The variables declared after the comma are too easy to miss.
+ /* Collect some stats. */ + size_t pages = 0; /* Number of hbin pages read. */ + size_t smallest_page = SIZE_MAX, largest_page = 0; + size_t blocks = 0; /* Total number of blocks found. */ + size_t smallest_block = SIZE_MAX, largest_block = 0, blocks_bytes = 0; + size_t used_blocks = 0; /* Total number of used blocks found. */ + size_t used_size = 0; /* Total size (bytes) of used blocks. */
Maybe it's just that I'm too used to never doing that, especially with initializers. ACK
On Thu, Jan 28, 2010 at 12:21:11PM +0100, Jim Meyering wrote:
Richard W.M. Jones wrote:
Subject: [PATCH 03/13] hivex: Collect more statistics about registries.
This looks fine. Mostly mechanical, like s/h->var/var/, ...
- filename, off, h->pages); + filename, off, pages);
... with type adjustments
- int32_t seg_len; + size_t seg_len;
However, I find the following hard to read: The variables declared after the comma are too easy to miss.
+ /* Collect some stats. */ + size_t pages = 0; /* Number of hbin pages read. */ + size_t smallest_page = SIZE_MAX, largest_page = 0; + size_t blocks = 0; /* Total number of blocks found. */ + size_t smallest_block = SIZE_MAX, largest_block = 0, blocks_bytes = 0; + size_t used_blocks = 0; /* Total number of used blocks found. */ + size_t used_size = 0; /* Total size (bytes) of used blocks. */
Maybe it's just that I'm too used to never doing that, especially with initializers.
Yes I agree they are a bit hard to see.
ACK
Thanks - I've now pushed 1 & 3. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
Richard W.M. Jones wrote:
Subject: [PATCH 04/13] hivex: page 'offset_next' field is really 'page_size'.
The documentation, as usual, is contradictory. However this field is definitely the page size in all observed registries. Furthermore the following field marked 'unknown' is always zero, although this contradicts what the sentinelchicken.com paper says.
Fun, eh? ;-) This looks fine. ACK
On Thu, Jan 28, 2010 at 12:26:31PM +0100, Jim Meyering wrote:
Richard W.M. Jones wrote:
Subject: [PATCH 04/13] hivex: page 'offset_next' field is really 'page_size'.
The documentation, as usual, is contradictory. However this field is definitely the page size in all observed registries. Furthermore the following field marked 'unknown' is always zero, although this contradicts what the sentinelchicken.com paper says.
Fun, eh? ;-)
No it's not! Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
Richard W.M. Jones wrote:
Subject: [PATCH 05/13] hivex: Move header checksum code into a function.
This function can be reused later. ... +static uint32_t +header_checksum (hive_h *h)
This parameter can/should be "const". Is there some reason it's not?
+{ + uint32_t *daddr = (uint32_t *) h->addr; + size_t i; + uint32_t sum = 0; + + for (i = 0; i < 0x1fc / 4; ++i) { + sum ^= le32toh (*daddr); + daddr++; + } + + return sum; +} + hive_h * hivex_open (const char *filename, int flags) { @@ -340,14 +355,7 @@ hivex_open (const char *filename, int flags) goto error;
/* Header checksum. */ - uint32_t *daddr = (uint32_t *) h->addr; - size_t i; - uint32_t sum = 0; - for (i = 0; i < 0x1fc / 4; ++i) { - sum ^= le32toh (*daddr); - daddr++; - } - + uint32_t sum = header_checksum (h); if (sum != le32toh (h->hdr->csum)) { fprintf (stderr, "hivex: %s: bad checksum in hive header\n", filename); errno = EINVAL;
On Thu, Jan 28, 2010 at 06:35:29PM +0100, Jim Meyering wrote:
Richard W.M. Jones wrote:
Subject: [PATCH 05/13] hivex: Move header checksum code into a function.
This function can be reused later. ... +static uint32_t +header_checksum (hive_h *h)
This parameter can/should be "const". Is there some reason it's not?
Yes that is correct. And I just pushed that change thinking it was so simple :-( I'll fix it with a const-correctness commit now. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
Richard W.M. Jones wrote:
Subject: [PATCH 06/13] hivex: Add value_any callback to the visitor.
The visitor currently contains lots of value_* callbacks, such as value_string which is called back when the value has type string.
This is fine but it makes it complicated to deal with the case where you just want to see 'a value', and don't care about its type.
The value_any callback allows visitors to see values generically. --- hivex/hivex.c | 178 +++++++++++++++++++++++++++++-------------------------- hivex/hivex.h | 1 + hivex/hivex.pod | 5 ++ 3 files changed, 101 insertions(+), 83 deletions(-)
diff --git a/hivex/hivex.c b/hivex/hivex.c index cb6fc62..81d217b 100644 --- a/hivex/hivex.c +++ b/hivex/hivex.c @@ -1421,114 +1421,126 @@ hivex__visit_node (hive_h *h, hive_node_h node, goto error; }
- switch (t) { - case hive_t_none: + if (vtor->value_any) { str = hivex_value_value (h, values[i], &t, &len); if (str == NULL) { ret = skip_bad ? 0 : -1; goto error; } - if (t != hive_t_none) { - ret = skip_bad ? 0 : -1; - goto error; - } - if (vtor->value_none && - vtor->value_none (h, opaque, node, values[i], t, len, key, str) == -1) + if (vtor->value_any (h, opaque, node, values[i], t, len, key, str) == -1) goto error; free (str); str = NULL; - break; - - case hive_t_string: - case hive_t_expand_string: - case hive_t_link: - str = hivex_value_string (h, values[i]); - if (str == NULL) { - if (errno != EILSEQ && errno != EINVAL) { + } + else { + switch (t) {
This looks fine. I found it best to review with "git log -B -w -p -1 SHA1" since the vast majority is just an indentation change: moving the current "switch" stmt into the new else block.
On Thu, Jan 28, 2010 at 03:21:50PM +0100, Jim Meyering wrote:
Richard W.M. Jones wrote:
Subject: [PATCH 06/13] hivex: Add value_any callback to the visitor.
The visitor currently contains lots of value_* callbacks, such as value_string which is called back when the value has type string.
This is fine but it makes it complicated to deal with the case where you just want to see 'a value', and don't care about its type.
The value_any callback allows visitors to see values generically. --- hivex/hivex.c | 178 +++++++++++++++++++++++++++++-------------------------- hivex/hivex.h | 1 + hivex/hivex.pod | 5 ++ 3 files changed, 101 insertions(+), 83 deletions(-)
diff --git a/hivex/hivex.c b/hivex/hivex.c index cb6fc62..81d217b 100644 --- a/hivex/hivex.c +++ b/hivex/hivex.c @@ -1421,114 +1421,126 @@ hivex__visit_node (hive_h *h, hive_node_h node, goto error; }
- switch (t) { - case hive_t_none: + if (vtor->value_any) { str = hivex_value_value (h, values[i], &t, &len); if (str == NULL) { ret = skip_bad ? 0 : -1; goto error; } - if (t != hive_t_none) { - ret = skip_bad ? 0 : -1; - goto error; - } - if (vtor->value_none && - vtor->value_none (h, opaque, node, values[i], t, len, key, str) == -1) + if (vtor->value_any (h, opaque, node, values[i], t, len, key, str) == -1) goto error; free (str); str = NULL; - break; - - case hive_t_string: - case hive_t_expand_string: - case hive_t_link: - str = hivex_value_string (h, values[i]); - if (str == NULL) { - if (errno != EILSEQ && errno != EINVAL) { + } + else { + switch (t) {
This looks fine. I found it best to review with "git log -B -w -p -1 SHA1" since the vast majority is just an indentation change: moving the current "switch" stmt into the new else block.
I was playing with git show --patience and other flags like that. The patience algorithm (sometimes) improves the output by keeping related code together. Is there a recommended way to get better patches for review from 'git format-patch'? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
Richard W.M. Jones wrote:
This looks fine. I found it best to review with "git log -B -w -p -1 SHA1" since the vast majority is just an indentation change: moving the current "switch" stmt into the new else block.
I was playing with git show --patience and other flags like that. The patience algorithm (sometimes) improves the output by keeping related code together. Is there a recommended way to get better patches for review from 'git format-patch'?
For indentation-changing change sets like that, I like to see "git diff -B -w" output. Sometimes, I use "-U N", too, using N large enough so the context includes some related part of the code. Note however, that if you use an option like -B or -w, you have to be careful to insert it, *indented*, after the "---" line and before the first diff, so that when someone tries to apply your patch with git-am that part is ignored. This would be important if it's in the middle of a large series like this. For a small patch set, just include it inline and warn that it should not be applied.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
Richard W.M. Jones wrote:
Subject: [PATCH 08/13] hivex: Clarify some more fields.
Taken from sentinelchicken.com documentation. --- hivex/hivex.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/hivex/hivex.c b/hivex/hivex.c index dfac896..1f5c08b 100644 --- a/hivex/hivex.c +++ b/hivex/hivex.c @@ -203,7 +203,8 @@ struct ntreg_nk_record { int32_t seg_len; /* length (always -ve because used) */ char id[2]; /* "nk" */ uint16_t flags; - char timestamp[12]; + char timestamp[8]; + char unknown0[4];
I wonder if it's nanoseconds... Obviously ok, since these members are not used and the struct size does not change.
uint32_t parent; /* offset of owner/parent */ uint32_t nr_subkeys; /* number of subkeys */ uint32_t unknown1; @@ -226,7 +227,7 @@ struct ntreg_lf_record { uint16_t nr_keys; /* number of keys in this record */ struct { uint32_t offset; /* offset of nk-record for this subkey */ - char name[4]; /* first 4 characters of subkey name */ + char hash[4]; /* hash of subkey name */ } keys[1]; } __attribute__((__packed__));
On Thu, Jan 28, 2010 at 03:26:19PM +0100, Jim Meyering wrote:
Richard W.M. Jones wrote:
Subject: [PATCH 08/13] hivex: Clarify some more fields.
Taken from sentinelchicken.com documentation. --- hivex/hivex.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/hivex/hivex.c b/hivex/hivex.c index dfac896..1f5c08b 100644 --- a/hivex/hivex.c +++ b/hivex/hivex.c @@ -203,7 +203,8 @@ struct ntreg_nk_record { int32_t seg_len; /* length (always -ve because used) */ char id[2]; /* "nk" */ uint16_t flags; - char timestamp[12]; + char timestamp[8]; + char unknown0[4];
I wonder if it's nanoseconds...
No it's much stranger than that. Tenths of microseconds since Jan 1 1601. http://support.microsoft.com/kb/167296 The following OCaml code successfully converts it to a time_t (from my reverse-engineering analysis prog): let nt_to_time_t t = let t = Int64.sub t 116444736000000000L in let t = Int64.div t 10000000L in Int64.to_float t Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
Richard W.M. Jones wrote:
On Thu, Jan 28, 2010 at 03:26:19PM +0100, Jim Meyering wrote:
Richard W.M. Jones wrote:
Subject: [PATCH 08/13] hivex: Clarify some more fields.
Taken from sentinelchicken.com documentation. --- hivex/hivex.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/hivex/hivex.c b/hivex/hivex.c index dfac896..1f5c08b 100644 --- a/hivex/hivex.c +++ b/hivex/hivex.c @@ -203,7 +203,8 @@ struct ntreg_nk_record { int32_t seg_len; /* length (always -ve because used) */ char id[2]; /* "nk" */ uint16_t flags; - char timestamp[12]; + char timestamp[8]; + char unknown0[4];
I wonder if it's nanoseconds...
No it's much stranger than that. Tenths of microseconds since Jan 1 1601.
I meant the "unknown0" member.
On Thu, Jan 28, 2010 at 03:48:44PM +0100, Jim Meyering wrote:
Richard W.M. Jones wrote:
On Thu, Jan 28, 2010 at 03:26:19PM +0100, Jim Meyering wrote:
Richard W.M. Jones wrote:
Subject: [PATCH 08/13] hivex: Clarify some more fields.
Taken from sentinelchicken.com documentation. --- hivex/hivex.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/hivex/hivex.c b/hivex/hivex.c index dfac896..1f5c08b 100644 --- a/hivex/hivex.c +++ b/hivex/hivex.c @@ -203,7 +203,8 @@ struct ntreg_nk_record { int32_t seg_len; /* length (always -ve because used) */ char id[2]; /* "nk" */ uint16_t flags; - char timestamp[12]; + char timestamp[8]; + char unknown0[4];
I wonder if it's nanoseconds...
No it's much stranger than that. Tenths of microseconds since Jan 1 1601.
I meant the "unknown0" member.
That particular field is always 0 in the files I have access to. The other "unknown" fields seem to be significant and that's what I'm trying to work out now. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
Richard W.M. Jones wrote: >>From 596de09265076de8fe4909f662e898159dfc3736 Mon Sep 17 00:00:00 2001 > This was missing. It only worked because we test on a little > endian platform. ... > - hive_node_h subkey = lf->keys[i].offset; > + hive_node_h subkey = le32toh (lf->keys[i].offset); Is there a convenient way to build just the hivex-related bits and test it independently of all the rest? Since it's relatively portable, that should be feasible to run on ppc or s390. That looks fine.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
Richard W.M. Jones wrote:
Subject: [PATCH 10/13] hivex: hive type in vk-record is an unsigned 32 bit int index d8e599b..4fe8175 100644 ... uint32_t data_len; uint32_t data_offset; /* pointer to the data (or data if inline) */ - hive_type data_type; /* type of the data */ + uint32_t data_type; /* type of the data */ uint16_t flags; /* bit 0 set => key name ASCII, bit 0 clr => key name UTF-16.
hive_type is an enum, so this should induce no size change. ACK.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v
Richard W.M. Jones wrote:
Subject: [PATCH 11/13] hivex: display bad block offset in hex
--- hivex/hivex.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/hivex/hivex.c b/hivex/hivex.c index 4fe8175..1eadee3 100644 --- a/hivex/hivex.c +++ b/hivex/hivex.c @@ -459,7 +459,7 @@ hivex_open (const char *filename, int flags) int used; seg_len = block_len (h, blkoff, &used); if (seg_len <= 4 || (seg_len & 3) != 0) { - fprintf (stderr, "hivex: %s: block size %d at %zu, bad registry\n", + fprintf (stderr, "hivex: %s: block size %d at 0x%zx, bad registry\n",
Obviously correct.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://et.redhat.com/~rjones/libguestfs/ See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html
Richard W.M. Jones wrote:
Subject: [PATCH 12/13] hivex: Display incorrect block size as unsigned in an error message.
--- hivex/hivex.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/hivex/hivex.c b/hivex/hivex.c index 1eadee3..4e67b55 100644 --- a/hivex/hivex.c +++ b/hivex/hivex.c @@ -459,7 +459,7 @@ hivex_open (const char *filename, int flags) int used; seg_len = block_len (h, blkoff, &used); if (seg_len <= 4 || (seg_len & 3) != 0) { - fprintf (stderr, "hivex: %s: block size %d at 0x%zx, bad registry\n", + fprintf (stderr, "hivex: %s: block size %" PRIu32 " at 0x%zx, bad registry\n", filename, le32toh (block->seg_len), blkoff);
That's fine, too.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
Richard W.M. Jones wrote:
Subject: [PATCH 13/13] hivex: Fix calculation of block size for vk data blocks. diff --git a/hivex/hivex.c b/hivex/hivex.c ... /* Check that the declared size isn't larger than the block its in. */ size_t blen = block_len (h, data_offset, NULL); - if (len > blen) { + if (len > blen - 4 /* subtract 4 for block header */) {
I presume there's no symbolic (e.g., sizeof-related) way to get that "4". ACK.
On Thu, Jan 28, 2010 at 06:32:47PM +0100, Jim Meyering wrote:
Richard W.M. Jones wrote:
Subject: [PATCH 13/13] hivex: Fix calculation of block size for vk data blocks. diff --git a/hivex/hivex.c b/hivex/hivex.c ... /* Check that the declared size isn't larger than the block its in. */ size_t blen = block_len (h, data_offset, NULL); - if (len > blen) { + if (len > blen - 4 /* subtract 4 for block header */) {
I presume there's no symbolic (e.g., sizeof-related) way to get that "4". ACK.
It's fairly well baked into the format. Microsoft didn't change the hive format even when they ported Windows to 64 bit, even though the registry is just a memory dump ... Oh this format is horrible. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
This final patch actually implements writing to hives. It is not complete yet because although it works as far as our tools are concerned, Windows ignores any new values added to a node, for reasons which we don't yet understand. Therefore I am continuing to reverse- engineer the hive format itself so that we fully understand all the fields. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
participants (2)
-
Jim Meyering -
Richard W.M. Jones