Problem with allocation of big files
by Nikita A Menkovich
Hello,
I found a strange problem with allocationg big files on drive.
For example in guestfish I allocate large disk image
for example:
$ guestfish
><fs> allocate test.img 20G
When an image allocating, there is a big slowdown of guest OSes,
launched on host machine, and on different drives.
For Linux guests with virtio drivers, there is no so big performance
penalty, but for FreeBSD guests with ide it seems like the OS locked.
Manipulating with ionice and niceness do not affect at all.
Manipulating with cfq, deadline schedulers also do not have any results.
OS Debian Squeeze
libvirtd (libvirt) 0.9.2
QEMU emulator version 0.14.0 (Debian 0.14.0+dfsg-5.1), Copyright (c)
2003-2008 Fabrice Bellard
guestfish 1.10.3 with some local patches, backported from main tree
Does anyone have ideas how to fix it?
--
Nikita A Menkovich
http://libc6.org/
JID: menkovich(a)gmail.com
13 years, 2 months
[PATCH 0/7] hivex + hivexml: Add byte runs for nodes and values
by Alex Nelson
This changeset adds byte run reporters for node and value metadata in the
hivexml program. This location reporting required several new ABI
functions, which required new ABI return types. One benefit to the byte
run functions is additional sanity checks, which have revealed new data
or parsing errors when run on M57 patents images. An example error:
Image: Charlie, 2009-12-11, available at <http://digitalcorpora.org/corpora/scenarios/m57-patents-scenario>.
hive: C:/WINDOWS/system32/config/SECURITY
Address 12624 is processed as a value, but it has a node signature.
Alex Nelson (7):
generator: Add new return type to ABI: RSize
hivex: Split value_key function into value_key and value_key_len
generator: Add new return types to ABI: RLenNode and RLenValue
hivex: Add metadata length functions for nodes and values
hivex: Add offset-&-length function for long value data
hivexml: Report attributes in values instead of text.
hivexml: Add byte run reporting functions
generator/generator.ml | 169 +++++++++++++++++++++++++++++++++++++++++++++++-
lib/hivex.c | 115 +++++++++++++++++++++++++++++---
xml/hivexml.c | 127 ++++++++++++++++++++++++++++++++----
3 files changed, 385 insertions(+), 26 deletions(-)
--
1.7.6
13 years, 2 months
Patch to build hivex lib on Windows
by Gillen Daniel
Hi
I'm just posting this here in case someone is interested in building
hivex on Windows (mingw32). The attached patch allows building the lib
but not the tools (hivexsh etc..) as there are some more problems to solve.
In short terms, this patch replaces file i/o functions and mmap(),
munmap() with their win32api pendants.
cu
--
Unix _IS_ user friendly - it's just
selective about who its friends are!
13 years, 2 months
[PATCH] hivexsh: Conditionally build for Mac OS X
by Alex Nelson
OS X lacks open_memstream, causing hivexsh to fail to build. This patch
defines HAVE_HIVEXSH, setting the only condition to open_memstream
existence.
Signed-off-by: Alex Nelson <ajnelson(a)cs.ucsc.edu>
---
Makefile.am | 6 +++++-
configure.ac | 4 ++++
2 files changed, 9 insertions(+), 1 deletions(-)
diff --git a/Makefile.am b/Makefile.am
index 8d69c55..dbab38a 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -20,7 +20,11 @@ ACLOCAL_AMFLAGS = -I m4
# Work around broken libtool.
export to_tool_file_cmd=func_convert_file_noop
-SUBDIRS = gnulib/lib generator lib images gnulib/tests sh xml po
+SUBDIRS = gnulib/lib generator lib images gnulib/tests xml po
+
+if HAVE_HIVEXSH
+SUBDIRS += sh
+endif
if HAVE_OCAML
SUBDIRS += ocaml
diff --git a/configure.ac b/configure.ac
index d7520e7..ce9dd48 100644
--- a/configure.ac
+++ b/configure.ac
@@ -135,6 +135,7 @@ AC_CHECK_HEADERS([byteswap.h endian.h libintl.h])
dnl Functions.
AC_CHECK_FUNCS([bindtextdomain])
+AC_CHECK_FUNCS([open_memstream])
dnl Check for pod2man and pod2text.
AC_CHECK_PROG([POD2MAN],[pod2man],[pod2man],[no])
@@ -173,6 +174,9 @@ PKG_CHECK_MODULES([LIBXML2], [libxml-2.0])
AC_SUBST([LIBXML2_CFLAGS])
AC_SUBST([LIBXML2_LIBS])
+dnl hivexsh depends on open_memstream, which is absent on OS X.
+AM_CONDITIONAL([HAVE_HIVEXSH],[test "x$HAVE_OPEN_MEMSTREAM" = "xyes"])
+
dnl Check for OCaml (optional, for OCaml bindings).
AC_PROG_OCAML
AC_PROG_FINDLIB
--
1.7.4.4
13 years, 2 months
[PATCH 0/3] hivex: Improve OS X support
by Alex Nelson
This patch series improves, but does not complete, OS X support for
hivex. There are several outstanding issues before hivexml and hivexsh
can run:
* hivexsh uses open_memstream, which doesn't exist in OS X as near as I
can see. Unfortunately, I'm inexperienced with autoconf, so I'm not
sure how to employ AC_CHECK_FUNCS([open_memstream]) to conditionally add
'sh' to the Makefile.am's SUBDIRS.
* The check for libxml fails, because PKG_CHECK_CONFIG doesn't exist on
OS X. I think it would be better to use AC_CHECK_FUNCS on some of the
libxml functions, but again, I'm not too hot with autoconf and automake.
* hivexml has the following run-time failure:
dyld: lazy symbol binding failed: Symbol not found: _iconv_open
Referenced from: /Users/alex/local/src/hivex/lib/.libs/libhivex.0.dylib
Expected in: flat namespace
dyld: Symbol not found: _iconv_open
Referenced from: /Users/alex/local/src/hivex/lib/.libs/libhivex.0.dylib
Expected in: flat namespace
Trace/BPT trap: 5
I have the vague idea of how to address the first two, but not the last.
I would appreciate any help on these. Gillen, you said earlier this
weekend you had taken a stab at compiling things in OS X. Did you have
to take these steps, too?
--Alex
Alex Nelson (3):
Mac OS X: Detect bindtextdomain
Mac OS X: setlocale function requires locale.h
Mac OS X: Run glibtoolize in absence of libtoolize
bootstrap | 2 +-
configure.ac | 3 +++
sh/hivexsh.c | 3 +++
xml/hivexml.c | 3 +++
4 files changed, 10 insertions(+), 1 deletions(-)
--
1.7.4.4
13 years, 2 months
Re: [Libguestfs] guestfish zerofree on LVM ?
by Richard W.M. Jones
On Tue, Sep 06, 2011 at 01:16:42PM -0400, Ben Clay wrote:
> Rich-
>
> Sorry for the delay, my box went down over the weekend.
>
> I have zerofree installed to /usr/sbin, but it's from the EPEL 5 repo when
> I'm using CentOS 6 x86_64. I was unable to find a zerofree rpm for
> CentOS/RHEL 6 x86_64. I manually installed the zerofree 1.0.1 rpm since yum
> wouldn't pick it up, and /usr/sbin/zerofree appears to run.
>
> However, in guestfish, it's missing:
>
> ><fs> available "zerofree"
> libguestfs: error: available: zerofree: group not available
Here is the code:
http://git.annexia.org/?p=libguestfs.git;a=blob;f=daemon/zerofree.c;h=f98...
As you can see, it is checking for a zerofree binary somewhere along
the $PATH inside the appliance. In the appliance,
PATH=/usr/sbin:/usr/bin:/sbin:/bin
> ><fs> debug ll /usr/sbin
[...]
> -rwxr-xr-x 1 root root 52992 Nov 12 2010 vipw
> -rwxr-xr-x 1 root root 17392 Jun 25 12:40 zdump
> -rwxr-xr-x 1 root root 50256 Jun 25 12:40 zic
Yup, not there.
> Here is my guestfish version:
>
> ]# guestfish version
> major: 1
> minor: 2
> release: 7
> extra:
>
> Does this version not have zerofree support? I checked the release notes
> and do not see it mentioned.
By the way, this is a really really old version of libguestfs ...
There should be a host file called something similar to
/usr/lib*/guestfs/initramfs.epel-5.i686.supermin.hostfiles
(it might have a slightly different name).
This file should contain a line:
./usr/sbin/zerofree
For example:
$ grep zerofree /usr/lib/guestfs/initramfs.epel-5.i686.supermin.hostfiles
./usr/sbin/zerofree
If that line is there, then /usr/sbin/zerofree on the host filesystem
should be copied into the appliance.
If it is not there, you could try adding that line. NB the initial dot.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
13 years, 2 months
[PATCH 6/7] hivexml: Report attributes in values instead of text.
by Alex Nelson
Reporting value data in attributes has two advantages:
* The output of hivexml breaks Python expat processing if binary data
makes it out. This was observed in Software hives.
* Not having child text makes room for child elements.
Signed-off-by: Alex Nelson <ajnelson(a)cs.ucsc.edu>
---
xml/hivexml.c | 25 ++++++++++++++++++++-----
1 files changed, 20 insertions(+), 5 deletions(-)
diff --git a/xml/hivexml.c b/xml/hivexml.c
index f29c80c..4789dbc 100644
--- a/xml/hivexml.c
+++ b/xml/hivexml.c
@@ -199,6 +199,7 @@ node_start (hive_h *h, void *writer_v, hive_node_h node, const char *name)
{
int64_t last_modified;
char *timebuf;
+ int ret = 0;
xmlTextWriterPtr writer = (xmlTextWriterPtr) writer_v;
XML_CHECK (xmlTextWriterStartElement, (writer, BAD_CAST "node"));
@@ -278,7 +279,9 @@ value_string (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value,
}
start_value (writer, key, type, NULL);
- XML_CHECK (xmlTextWriterWriteString, (writer, BAD_CAST str));
+ XML_CHECK (xmlTextWriterStartAttribute, (writer, BAD_CAST "value"));
+ XML_CHECK (xmlTextWriterWriteString, (writer, str));
+ XML_CHECK (xmlTextWriterEndAttribute, (writer));
end_value (writer);
return 0;
}
@@ -332,7 +335,9 @@ value_string_invalid_utf16 (hive_h *h, void *writer_v, hive_node_h node,
}
start_value (writer, key, type, "base64");
+ XML_CHECK (xmlTextWriterStartAttribute, (writer, BAD_CAST "value"));
XML_CHECK (xmlTextWriterWriteBase64, (writer, str, 0, len));
+ XML_CHECK (xmlTextWriterEndAttribute, (writer));
end_value (writer);
return 0;
@@ -344,7 +349,7 @@ value_dword (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value,
{
xmlTextWriterPtr writer = (xmlTextWriterPtr) writer_v;
start_value (writer, key, "int32", NULL);
- XML_CHECK (xmlTextWriterWriteFormatString, (writer, "%" PRIi32, v));
+ XML_CHECK (xmlTextWriterWriteFormatAttribute, (writer, BAD_CAST "value", "%" PRIi32, v));
end_value (writer);
return 0;
}
@@ -355,7 +360,7 @@ value_qword (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value,
{
xmlTextWriterPtr writer = (xmlTextWriterPtr) writer_v;
start_value (writer, key, "int64", NULL);
- XML_CHECK (xmlTextWriterWriteFormatString, (writer, "%" PRIi64, v));
+ XML_CHECK (xmlTextWriterWriteFormatAttribute, (writer, "value", "%" PRIi64, v));
end_value (writer);
return 0;
}
@@ -366,7 +371,9 @@ value_binary (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value,
{
xmlTextWriterPtr writer = (xmlTextWriterPtr) writer_v;
start_value (writer, key, "binary", "base64");
+ XML_CHECK (xmlTextWriterStartAttribute, (writer, BAD_CAST "value"));
XML_CHECK (xmlTextWriterWriteBase64, (writer, v, 0, len));
+ XML_CHECK (xmlTextWriterEndAttribute, (writer));
end_value (writer);
return 0;
}
@@ -377,7 +384,11 @@ value_none (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value,
{
xmlTextWriterPtr writer = (xmlTextWriterPtr) writer_v;
start_value (writer, key, "none", "base64");
- if (len > 0) XML_CHECK (xmlTextWriterWriteBase64, (writer, v, 0, len));
+ if (len > 0) {
+ XML_CHECK (xmlTextWriterStartAttribute, (writer, BAD_CAST "value"));
+ XML_CHECK (xmlTextWriterWriteBase64, (writer, v, 0, len));
+ XML_CHECK (xmlTextWriterEndAttribute, (writer));
+ }
end_value (writer);
return 0;
}
@@ -410,7 +421,11 @@ value_other (hive_h *h, void *writer_v, hive_node_h node, hive_value_h value,
}
start_value (writer, key, type, "base64");
- if (len > 0) XML_CHECK (xmlTextWriterWriteBase64, (writer, v, 0, len));
+ if (len > 0) {
+ XML_CHECK (xmlTextWriterStartAttribute, (writer, BAD_CAST "value"));
+ XML_CHECK (xmlTextWriterWriteBase64, (writer, v, 0, len));
+ XML_CHECK (xmlTextWriterEndAttribute, (writer));
+ }
end_value (writer);
return 0;
--
1.7.6
13 years, 2 months
[PATCH 5/7] hivex: Add offset-&-length function for long value data
by Alex Nelson
This patch adds value_data_cell_offset to the hivex ABI, to report the
hive space used for long value data.
Signed-off-by: Alex Nelson <ajnelson(a)cs.ucsc.edu>
---
generator/generator.ml | 11 +++++++++++
lib/hivex.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 56 insertions(+), 0 deletions(-)
diff --git a/generator/generator.ml b/generator/generator.ml
index 5a71930..b607421 100755
--- a/generator/generator.ml
+++ b/generator/generator.ml
@@ -276,6 +276,17 @@ and sets errno on error.";
Return the length of the value data structure. Returns 0
and sets errno on error.";
+ "value_data_cell_offset", (RLenValue, [AHive; AValue "val"]),
+ "return the offset and length of a value data cell",
+ "\
+Return the offset and length of the value's data cell, not value cell.
+E.g. if the value were \"foobar\" then the length would be 6, and the
+offset would be to a value data cell, which is the data prefixed with
+4 bytes describing the size. If the length is not greater than 4,
+then 0 is returned, as the data are inline in the value.
+
+Returns 0 and sets errno on error.";
+
"value_value", (RLenTypeVal, [AHive; AValue "val"]),
"return data length, data type and data of a value",
"\
diff --git a/lib/hivex.c b/lib/hivex.c
index d8ffa63..04ceed3 100644
--- a/lib/hivex.c
+++ b/lib/hivex.c
@@ -1294,6 +1294,51 @@ hivex_value_type (hive_h *h, hive_value_h value, hive_type *t, size_t *len)
return 0;
}
+hive_value_h
+hivex_value_data_cell_offset (hive_h *h, hive_value_h value, size_t *len)
+{
+ if (!IS_VALID_BLOCK (h, value) || !BLOCK_ID_EQ (h, value, "vk")) {
+ errno = EINVAL;
+ return 0;
+ }
+
+ struct ntreg_vk_record *vk = (struct ntreg_vk_record *) (h->addr + value);
+
+ size_t data_len;
+ int is_inline;
+
+ data_len = le32toh (vk->data_len);
+ is_inline = !!(data_len & 0x80000000);
+ data_len &= 0x7fffffff;
+
+ if (is_inline && data_len > 4) {
+ errno = ENOTSUP;
+ return 0;
+ }
+
+ if (is_inline) {
+ /* There is no other location for the value data. */
+ if (len)
+ *len = 0;
+ return 0;
+ } else {
+ if (len)
+ *len = data_len + 4; /* Include 4 header length bytes */
+ }
+
+ size_t data_offset = le32toh (vk->data_offset);
+ data_offset += 0x1000; /* Add 0x1000 because everything's off by 4KiB */
+ if (!IS_VALID_BLOCK (h, data_offset)) {
+ if (h->msglvl >= 2)
+ fprintf (stderr, "hivex_value_data_cell_offset: returning EFAULT because data "
+ "offset is not a valid block (0x%zx)\n",
+ data_offset);
+ errno = EFAULT;
+ return 0;
+ }
+ return data_offset;
+}
+
char *
hivex_value_value (hive_h *h, hive_value_h value,
hive_type *t_rtn, size_t *len_rtn)
--
1.7.6
13 years, 2 months
[PATCH 4/7] hivex: Add metadata length functions for nodes and values
by Alex Nelson
This patch adds hivex_node_struct_length and hivex_value_struct_length
to the hivex ABI, to report the amount of hive space used for each
stored structure.
Signed-off-by: Alex Nelson <ajnelson(a)cs.ucsc.edu>
---
generator/generator.ml | 12 ++++++++++++
lib/hivex.c | 33 +++++++++++++++++++++++++++++++++
2 files changed, 45 insertions(+), 0 deletions(-)
diff --git a/generator/generator.ml b/generator/generator.ml
index 15ebc44..5a71930 100755
--- a/generator/generator.ml
+++ b/generator/generator.ml
@@ -264,6 +264,18 @@ information, and the value itself. Also, C<hivex_value_*> functions
below which can be used to return the value in a more useful form when
you know the type in advance.";
+ "node_struct_length", (RSize, [AHive; ANode "node"]),
+ "return the length of a node",
+ "\
+Return the length of the node data structure. Returns 0
+and sets errno on error.";
+
+ "value_struct_length", (RSize, [AHive; AValue "val"]),
+ "return the length of a value data structure",
+ "\
+Return the length of the value data structure. Returns 0
+and sets errno on error.";
+
"value_value", (RLenTypeVal, [AHive; AValue "val"]),
"return data length, data type and data of a value",
"\
diff --git a/lib/hivex.c b/lib/hivex.c
index 61177d3..d8ffa63 100644
--- a/lib/hivex.c
+++ b/lib/hivex.c
@@ -585,6 +585,30 @@ hivex_root (hive_h *h)
return ret;
}
+size_t
+hivex_node_struct_length (hive_h *h, hive_node_h node)
+{
+ if (!IS_VALID_BLOCK (h, node) || !BLOCK_ID_EQ (h, node, "nk")) {
+ errno = EINVAL;
+ return 0;
+ }
+
+ struct ntreg_nk_record *nk = (struct ntreg_nk_record *) (h->addr + node);
+ size_t name_len = le16toh (nk->name_len);
+ /* -1 to avoid double-counting the first name character */
+ size_t ret = name_len + sizeof (struct ntreg_nk_record) - 1;
+ int used;
+ size_t seg_len = block_len (h, node, &used);
+ if (ret > seg_len) {
+ if (h->msglvl >= 2)
+ fprintf (stderr, "hivex_node_struct_length: returning EFAULT because"
+ " node name is too long (%zu, %zu)\n", name_len, seg_len);
+ errno = EFAULT;
+ return 0;
+ }
+ return ret;
+}
+
char *
hivex_node_name (hive_h *h, hive_node_h node)
{
@@ -1190,6 +1214,15 @@ hivex_node_get_value (hive_h *h, hive_node_h node, const char *key)
}
size_t
+hivex_value_struct_length (hive_h *h, hive_value_h value) {
+ size_t key_len = hivex_value_key_len (h, value);
+ if (errno)
+ return 0;
+ /* -1 to avoid double-counting the first name character */
+ return key_len + sizeof (struct ntreg_vk_record) - 1;
+}
+
+size_t
hivex_value_key_len (hive_h *h, hive_value_h value)
{
if (!IS_VALID_BLOCK (h, value) || !BLOCK_ID_EQ (h, value, "vk")) {
--
1.7.6
13 years, 2 months
