September 2011 - Libguestfs - libguestfs List Archives

Problem with allocation of big files
by Nikita A Menkovich 20 Sep '11

20 Sep '11

Hello, I found a strange problem with allocationg big files on drive. For example in guestfish I allocate large disk image for example: $ guestfish ><fs> allocate test.img 20G When an image allocating, there is a big slowdown of guest OSes, launched on host machine, and on different drives. For Linux guests with virtio drivers, there is no so big performance penalty, but for FreeBSD guests with ide it seems like the OS locked. Manipulating with ionice and niceness do not affect at all. Manipulating with cfq, deadline schedulers also do not have any results. OS Debian Squeeze libvirtd (libvirt) 0.9.2 QEMU emulator version 0.14.0 (Debian 0.14.0+dfsg-5.1), Copyright (c) 2003-2008 Fabrice Bellard guestfish 1.10.3 with some local patches, backported from main tree Does anyone have ideas how to fix it? -- Nikita A Menkovich http://libc6.org/ JID: menkovich(a)gmail.com

2 1

[PATCH 0/7] hivex + hivexml: Add byte runs for nodes and values
by Alex Nelson 17 Sep '11

17 Sep '11

This changeset adds byte run reporters for node and value metadata in the hivexml program. This location reporting required several new ABI functions, which required new ABI return types. One benefit to the byte run functions is additional sanity checks, which have revealed new data or parsing errors when run on M57 patents images. An example error: Image: Charlie, 2009-12-11, available at <http://digitalcorpora.org/corpora/scenarios/m57-patents-scenario>. hive: C:/WINDOWS/system32/config/SECURITY Address 12624 is processed as a value, but it has a node signature. Alex Nelson (7): generator: Add new return type to ABI: RSize hivex: Split value_key function into value_key and value_key_len generator: Add new return types to ABI: RLenNode and RLenValue hivex: Add metadata length functions for nodes and values hivex: Add offset-&-length function for long value data hivexml: Report attributes in values instead of text. hivexml: Add byte run reporting functions generator/generator.ml | 169 +++++++++++++++++++++++++++++++++++++++++++++++- lib/hivex.c | 115 +++++++++++++++++++++++++++++--- xml/hivexml.c | 127 ++++++++++++++++++++++++++++++++---- 3 files changed, 385 insertions(+), 26 deletions(-) -- 1.7.6

2 4

[hivex] RPMs for OCaml and Perl bindings
by Alex Nelson 11 Sep '11

11 Sep '11

Hi all, Which Yum packages are needed in a Fedora 15 environment to have the configure script build the Perl and OCaml bindings? --Alex

2 3

Patch to build hivex lib on Windows
by Gillen Daniel 07 Sep '11

07 Sep '11

Hi I'm just posting this here in case someone is interested in building hivex on Windows (mingw32). The attached patch allows building the lib but not the tools (hivexsh etc..) as there are some more problems to solve. In short terms, this patch replaces file i/o functions and mmap(), munmap() with their win32api pendants. cu -- Unix _IS_ user friendly - it's just selective about who its friends are!

3 15

[PATCH] hivexsh: Conditionally build for Mac OS X
by Alex Nelson 07 Sep '11

07 Sep '11

OS X lacks open_memstream, causing hivexsh to fail to build. This patch defines HAVE_HIVEXSH, setting the only condition to open_memstream existence. Signed-off-by: Alex Nelson <ajnelson(a)cs.ucsc.edu> --- Makefile.am | 6 +++++- configure.ac | 4 ++++ 2 files changed, 9 insertions(+), 1 deletions(-) diff --git a/Makefile.am b/Makefile.am index 8d69c55..dbab38a 100644 --- a/Makefile.am +++ b/Makefile.am @@ -20,7 +20,11 @@ ACLOCAL_AMFLAGS = -I m4 # Work around broken libtool. export to_tool_file_cmd=func_convert_file_noop -SUBDIRS = gnulib/lib generator lib images gnulib/tests sh xml po +SUBDIRS = gnulib/lib generator lib images gnulib/tests xml po + +if HAVE_HIVEXSH +SUBDIRS += sh +endif if HAVE_OCAML SUBDIRS += ocaml diff --git a/configure.ac b/configure.ac index d7520e7..ce9dd48 100644 --- a/configure.ac +++ b/configure.ac @@ -135,6 +135,7 @@ AC_CHECK_HEADERS([byteswap.h endian.h libintl.h]) dnl Functions. AC_CHECK_FUNCS([bindtextdomain]) +AC_CHECK_FUNCS([open_memstream]) dnl Check for pod2man and pod2text. AC_CHECK_PROG([POD2MAN],[pod2man],[pod2man],[no]) @@ -173,6 +174,9 @@ PKG_CHECK_MODULES([LIBXML2], [libxml-2.0]) AC_SUBST([LIBXML2_CFLAGS]) AC_SUBST([LIBXML2_LIBS]) +dnl hivexsh depends on open_memstream, which is absent on OS X. +AM_CONDITIONAL([HAVE_HIVEXSH],[test "x$HAVE_OPEN_MEMSTREAM" = "xyes"]) + dnl Check for OCaml (optional, for OCaml bindings). AC_PROG_OCAML AC_PROG_FINDLIB -- 1.7.4.4

2 1

[PATCH 0/3] hivex: Improve OS X support
by Alex Nelson 07 Sep '11

07 Sep '11

This patch series improves, but does not complete, OS X support for hivex. There are several outstanding issues before hivexml and hivexsh can run: * hivexsh uses open_memstream, which doesn't exist in OS X as near as I can see. Unfortunately, I'm inexperienced with autoconf, so I'm not sure how to employ AC_CHECK_FUNCS([open_memstream]) to conditionally add 'sh' to the Makefile.am's SUBDIRS. * The check for libxml fails, because PKG_CHECK_CONFIG doesn't exist on OS X. I think it would be better to use AC_CHECK_FUNCS on some of the libxml functions, but again, I'm not too hot with autoconf and automake. * hivexml has the following run-time failure: dyld: lazy symbol binding failed: Symbol not found: _iconv_open Referenced from: /Users/alex/local/src/hivex/lib/.libs/libhivex.0.dylib Expected in: flat namespace dyld: Symbol not found: _iconv_open Referenced from: /Users/alex/local/src/hivex/lib/.libs/libhivex.0.dylib Expected in: flat namespace Trace/BPT trap: 5 I have the vague idea of how to address the first two, but not the last. I would appreciate any help on these. Gillen, you said earlier this weekend you had taken a stab at compiling things in OS X. Did you have to take these steps, too? --Alex Alex Nelson (3): Mac OS X: Detect bindtextdomain Mac OS X: setlocale function requires locale.h Mac OS X: Run glibtoolize in absence of libtoolize bootstrap | 2 +- configure.ac | 3 +++ sh/hivexsh.c | 3 +++ xml/hivexml.c | 3 +++ 4 files changed, 10 insertions(+), 1 deletions(-) -- 1.7.4.4

3 5

Re: [Libguestfs] guestfish zerofree on LVM ?
by Richard W.M. Jones 06 Sep '11

06 Sep '11

On Tue, Sep 06, 2011 at 01:16:42PM -0400, Ben Clay wrote: > Rich- > > Sorry for the delay, my box went down over the weekend. > > I have zerofree installed to /usr/sbin, but it's from the EPEL 5 repo when > I'm using CentOS 6 x86_64. I was unable to find a zerofree rpm for > CentOS/RHEL 6 x86_64. I manually installed the zerofree 1.0.1 rpm since yum > wouldn't pick it up, and /usr/sbin/zerofree appears to run. > > However, in guestfish, it's missing: > > ><fs> available "zerofree" > libguestfs: error: available: zerofree: group not available Here is the code: http://git.annexia.org/?p=libguestfs.git;a=blob;f=daemon/zerofree.c;h=f98c6… As you can see, it is checking for a zerofree binary somewhere along the $PATH inside the appliance. In the appliance, PATH=/usr/sbin:/usr/bin:/sbin:/bin > ><fs> debug ll /usr/sbin [...] > -rwxr-xr-x 1 root root 52992 Nov 12 2010 vipw > -rwxr-xr-x 1 root root 17392 Jun 25 12:40 zdump > -rwxr-xr-x 1 root root 50256 Jun 25 12:40 zic Yup, not there. > Here is my guestfish version: > > ]# guestfish version > major: 1 > minor: 2 > release: 7 > extra: > > Does this version not have zerofree support? I checked the release notes > and do not see it mentioned. By the way, this is a really really old version of libguestfs ... There should be a host file called something similar to /usr/lib*/guestfs/initramfs.epel-5.i686.supermin.hostfiles (it might have a slightly different name). This file should contain a line: ./usr/sbin/zerofree For example: $ grep zerofree /usr/lib/guestfs/initramfs.epel-5.i686.supermin.hostfiles ./usr/sbin/zerofree If that line is there, then /usr/sbin/zerofree on the host filesystem should be copied into the appliance. If it is not there, you could try adding that line. NB the initial dot. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

1 0

[PATCH 6/7] hivexml: Report attributes in values instead of text.
by Alex Nelson 06 Sep '11

06 Sep '11

Reporting value data in attributes has two advantages: * The output of hivexml breaks Python expat processing if binary data makes it out. This was observed in Software hives. * Not having child text makes room for child elements. Signed-off-by: Alex Nelson <ajnelson(a)cs.ucsc.edu> --- xml/hivexml.c | 25 ++++++++++++++++++++----- 1 files changed, 20 insertions(+), 5 deletions(-) diff --git a/xml/hivexml.c b/xml/hivexml.c index f29c80c..4789dbc 100644 --- a/xml/hivexml.c +++ b/xml/hivexml.c @@ -199,6 +199,7 @@ node_start (hive_h *h, void *writer_v, hive_node_h node, const char *name) { int64_t last_modified; char *timebuf; + int ret = 0; xmlTextWriterPtr writer = (xmlTextWriterPtr) writer_v; XML_CHECK (xmlTextWriterStartElement, (writer, BAD_CAST "node")); @@ -278,7 +279,9 @@ value_string (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value, } start_value (writer, key, type, NULL); - XML_CHECK (xmlTextWriterWriteString, (writer, BAD_CAST str)); + XML_CHECK (xmlTextWriterStartAttribute, (writer, BAD_CAST "value")); + XML_CHECK (xmlTextWriterWriteString, (writer, str)); + XML_CHECK (xmlTextWriterEndAttribute, (writer)); end_value (writer); return 0; } @@ -332,7 +335,9 @@ value_string_invalid_utf16 (hive_h *h, void *writer_v, hive_node_h node, } start_value (writer, key, type, "base64"); + XML_CHECK (xmlTextWriterStartAttribute, (writer, BAD_CAST "value")); XML_CHECK (xmlTextWriterWriteBase64, (writer, str, 0, len)); + XML_CHECK (xmlTextWriterEndAttribute, (writer)); end_value (writer); return 0; @@ -344,7 +349,7 @@ value_dword (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value, { xmlTextWriterPtr writer = (xmlTextWriterPtr) writer_v; start_value (writer, key, "int32", NULL); - XML_CHECK (xmlTextWriterWriteFormatString, (writer, "%" PRIi32, v)); + XML_CHECK (xmlTextWriterWriteFormatAttribute, (writer, BAD_CAST "value", "%" PRIi32, v)); end_value (writer); return 0; } @@ -355,7 +360,7 @@ value_qword (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value, { xmlTextWriterPtr writer = (xmlTextWriterPtr) writer_v; start_value (writer, key, "int64", NULL); - XML_CHECK (xmlTextWriterWriteFormatString, (writer, "%" PRIi64, v)); + XML_CHECK (xmlTextWriterWriteFormatAttribute, (writer, "value", "%" PRIi64, v)); end_value (writer); return 0; } @@ -366,7 +371,9 @@ value_binary (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value, { xmlTextWriterPtr writer = (xmlTextWriterPtr) writer_v; start_value (writer, key, "binary", "base64"); + XML_CHECK (xmlTextWriterStartAttribute, (writer, BAD_CAST "value")); XML_CHECK (xmlTextWriterWriteBase64, (writer, v, 0, len)); + XML_CHECK (xmlTextWriterEndAttribute, (writer)); end_value (writer); return 0; } @@ -377,7 +384,11 @@ value_none (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value, { xmlTextWriterPtr writer = (xmlTextWriterPtr) writer_v; start_value (writer, key, "none", "base64"); - if (len > 0) XML_CHECK (xmlTextWriterWriteBase64, (writer, v, 0, len)); + if (len > 0) { + XML_CHECK (xmlTextWriterStartAttribute, (writer, BAD_CAST "value")); + XML_CHECK (xmlTextWriterWriteBase64, (writer, v, 0, len)); + XML_CHECK (xmlTextWriterEndAttribute, (writer)); + } end_value (writer); return 0; } @@ -410,7 +421,11 @@ value_other (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value, } start_value (writer, key, type, "base64"); - if (len > 0) XML_CHECK (xmlTextWriterWriteBase64, (writer, v, 0, len)); + if (len > 0) { + XML_CHECK (xmlTextWriterStartAttribute, (writer, BAD_CAST "value")); + XML_CHECK (xmlTextWriterWriteBase64, (writer, v, 0, len)); + XML_CHECK (xmlTextWriterEndAttribute, (writer)); + } end_value (writer); return 0; -- 1.7.6

2 1

[PATCH 5/7] hivex: Add offset-&-length function for long value data
by Alex Nelson 06 Sep '11

06 Sep '11

This patch adds value_data_cell_offset to the hivex ABI, to report the hive space used for long value data. Signed-off-by: Alex Nelson <ajnelson(a)cs.ucsc.edu> --- generator/generator.ml | 11 +++++++++++ lib/hivex.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 0 deletions(-) diff --git a/generator/generator.ml b/generator/generator.ml index 5a71930..b607421 100755 --- a/generator/generator.ml +++ b/generator/generator.ml @@ -276,6 +276,17 @@ and sets errno on error."; Return the length of the value data structure. Returns 0 and sets errno on error."; + "value_data_cell_offset", (RLenValue, [AHive; AValue "val"]), + "return the offset and length of a value data cell", + "\ +Return the offset and length of the value's data cell, not value cell. +E.g. if the value were \"foobar\" then the length would be 6, and the +offset would be to a value data cell, which is the data prefixed with +4 bytes describing the size. If the length is not greater than 4, +then 0 is returned, as the data are inline in the value. + +Returns 0 and sets errno on error."; + "value_value", (RLenTypeVal, [AHive; AValue "val"]), "return data length, data type and data of a value", "\ diff --git a/lib/hivex.c b/lib/hivex.c index d8ffa63..04ceed3 100644 --- a/lib/hivex.c +++ b/lib/hivex.c @@ -1294,6 +1294,51 @@ hivex_value_type (hive_h *h, hive_value_h value, hive_type *t, size_t *len) return 0; } +hive_value_h +hivex_value_data_cell_offset (hive_h *h, hive_value_h value, size_t *len) +{ + if (!IS_VALID_BLOCK (h, value) || !BLOCK_ID_EQ (h, value, "vk")) { + errno = EINVAL; + return 0; + } + + struct ntreg_vk_record *vk = (struct ntreg_vk_record *) (h->addr + value); + + size_t data_len; + int is_inline; + + data_len = le32toh (vk->data_len); + is_inline = !!(data_len & 0x80000000); + data_len &= 0x7fffffff; + + if (is_inline && data_len > 4) { + errno = ENOTSUP; + return 0; + } + + if (is_inline) { + /* There is no other location for the value data. */ + if (len) + *len = 0; + return 0; + } else { + if (len) + *len = data_len + 4; /* Include 4 header length bytes */ + } + + size_t data_offset = le32toh (vk->data_offset); + data_offset += 0x1000; /* Add 0x1000 because everything's off by 4KiB */ + if (!IS_VALID_BLOCK (h, data_offset)) { + if (h->msglvl >= 2) + fprintf (stderr, "hivex_value_data_cell_offset: returning EFAULT because data " + "offset is not a valid block (0x%zx)\n", + data_offset); + errno = EFAULT; + return 0; + } + return data_offset; +} + char * hivex_value_value (hive_h *h, hive_value_h value, hive_type *t_rtn, size_t *len_rtn) -- 1.7.6

2 1

[PATCH 4/7] hivex: Add metadata length functions for nodes and values
by Alex Nelson 06 Sep '11

06 Sep '11

This patch adds hivex_node_struct_length and hivex_value_struct_length to the hivex ABI, to report the amount of hive space used for each stored structure. Signed-off-by: Alex Nelson <ajnelson(a)cs.ucsc.edu> --- generator/generator.ml | 12 ++++++++++++ lib/hivex.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 0 deletions(-) diff --git a/generator/generator.ml b/generator/generator.ml index 15ebc44..5a71930 100755 --- a/generator/generator.ml +++ b/generator/generator.ml @@ -264,6 +264,18 @@ information, and the value itself. Also, C<hivex_value_*> functions below which can be used to return the value in a more useful form when you know the type in advance."; + "node_struct_length", (RSize, [AHive; ANode "node"]), + "return the length of a node", + "\ +Return the length of the node data structure. Returns 0 +and sets errno on error."; + + "value_struct_length", (RSize, [AHive; AValue "val"]), + "return the length of a value data structure", + "\ +Return the length of the value data structure. Returns 0 +and sets errno on error."; + "value_value", (RLenTypeVal, [AHive; AValue "val"]), "return data length, data type and data of a value", "\ diff --git a/lib/hivex.c b/lib/hivex.c index 61177d3..d8ffa63 100644 --- a/lib/hivex.c +++ b/lib/hivex.c @@ -585,6 +585,30 @@ hivex_root (hive_h *h) return ret; } +size_t +hivex_node_struct_length (hive_h *h, hive_node_h node) +{ + if (!IS_VALID_BLOCK (h, node) || !BLOCK_ID_EQ (h, node, "nk")) { + errno = EINVAL; + return 0; + } + + struct ntreg_nk_record *nk = (struct ntreg_nk_record *) (h->addr + node); + size_t name_len = le16toh (nk->name_len); + /* -1 to avoid double-counting the first name character */ + size_t ret = name_len + sizeof (struct ntreg_nk_record) - 1; + int used; + size_t seg_len = block_len (h, node, &used); + if (ret > seg_len) { + if (h->msglvl >= 2) + fprintf (stderr, "hivex_node_struct_length: returning EFAULT because" + " node name is too long (%zu, %zu)\n", name_len, seg_len); + errno = EFAULT; + return 0; + } + return ret; +} + char * hivex_node_name (hive_h *h, hive_node_h node) { @@ -1190,6 +1214,15 @@ hivex_node_get_value (hive_h *h, hive_node_h node, const char *key) } size_t +hivex_value_struct_length (hive_h *h, hive_value_h value) { + size_t key_len = hivex_value_key_len (h, value); + if (errno) + return 0; + /* -1 to avoid double-counting the first name character */ + return key_len + sizeof (struct ntreg_vk_record) - 1; +} + +size_t hivex_value_key_len (hive_h *h, hive_value_h value) { if (!IS_VALID_BLOCK (h, value) || !BLOCK_ID_EQ (h, value, "vk")) { -- 1.7.6

2 1