January 2014 - Libguestfs - libguestfs List Archives

[PATCH] builder: small code simplification
by Pino Toscano 22 Jan '14

22 Jan '14

No actual behaviour changes, just remove extra match statements. --- builder/list_entries.ml | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/builder/list_entries.ml b/builder/list_entries.ml index 742e43b..0d3d9b2 100644 --- a/builder/list_entries.ml +++ b/builder/list_entries.ml @@ -98,13 +98,11 @@ and list_entries_json ~sources index = | c -> String.make 1 c) done; !res in - let json_optional_printf_string key value = - match value with + let json_optional_printf_string key = function | None -> () | Some str -> printf " \"%s\": \"%s\",\n" key (json_string_escape str) in - let json_optional_printf_int64 key value = - match value with + let json_optional_printf_int64 key = function | None -> () | Some n -> printf " \"%s\": \"%Ld\",\n" key n in -- 1.8.3.1

2 1

[PATCH] builder: fix small regression in subkey parsing
by Pino Toscano 22 Jan '14

22 Jan '14

Introduced in 5cbdf35d651b6c730d62d9af4876039faa122efc, it caused the first character of the value to be skipped if the key has a subkey. --- builder/index-scan.l | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builder/index-scan.l b/builder/index-scan.l index 7a9618f..e43f82e 100644 --- a/builder/index-scan.l +++ b/builder/index-scan.l @@ -65,7 +65,7 @@ extern void yyerror (const char *); if (yytext[i] == '[') { size_t j = strcspn (yytext+i+1, "]"); yylval.field->subkey = strndup (yytext+i+1, j); - i += 1+j+2; + i += 1+j+1; } else { yylval.field->subkey = NULL; } -- 1.8.3.1

2 1

Re: [Libguestfs] [libguestfs] remove non-portable chmod from libtool-kill-dependency_libs.sh (#10)
by Richard W.M. Jones 21 Jan '14

21 Jan '14

Attaching the patch to the mailing list. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#)

1 1

virt-builder & virt-sysprep: Avoiding SELinux relabelling
by Richard W.M. Jones 21 Jan '14

21 Jan '14

A common problem that people have with virt-builder and virt-sysprep is which guests that use SELinux, like Fedora and RHEL. In both cases we touch /.autorelabel in the guest, which means the guest has to reboot once during its first boot. Recap: SELinux file labels -------------------------- SELinux requires that files have labels. Access to a file is controlled by the label on that file. For example: $ ls -lZ /etc/passwd -rw-r--r--. root root system_u:object_r:passwd_file_t:s0 /etc/passwd <----------- label -----------> The SELinux policy allows (eg) /usr/sbin/passwd to write to this file, based on the process label and the target file label. (The path "/etc/passwd" is not considered.) On a running system, files created from scratch are labelled by a daemon (restorecond, I think). Also they could be labelled by programs that create files such as 'rpm'. The label is actually stored in an extended attribute (xattr): $ getfattr -d --match=.* /etc/passwd getfattr: Removing leading '/' from absolute path names # file: etc/passwd security.selinux="system_u:object_r:passwd_file_t:s0" Libguestfs & SELinux labelling ------------------------------ Unfortunately SELinux requires a policy to be loaded into the kernel in order to know how to label files. There is no default labelling scheme for files, it depends entirely on the policy. Since the policy is guest specific, libguestfs cannot really load it (for a start, it wouldn't necessarily know where to load it from, and it might not be safe to load an untrusted policy). However libguestfs can do one of four things: (1) It can set the security.selinux xattr to some value. It has to know or be able to guess what value to set it to. Note there is no policy, so we have to infer this based on other things we know about the guest such as inspection data. (2) It can copy the security.selinux xattr from another file. (3) It can update the file in such a way that the xattr doesn't change, eg. appending to the file. (4) It can touch '/.autorelabel' which causes an SELinux enabled guest to do a full filesystem relabel at first boot (followed by a reboot). Virt-sysprep labelling ---------------------- As you may have guessed, currently virt-sysprep takes the easy option (4) above. Virt-sysprep tracks if any file has been created, and if so and the guest looks like an SELinux distro, it touches '/.autorelabel'. Since the 'hostname' operation is (a) enabled by default and (b) always creates the hostname file with default content even if the 'virt-sysprep --hostname' option was not used, almost any SELinux-enabled guest that virt-sysprep touches is going to have /.autorelabel at the end. Virt-builder labelling ---------------------- Virt-builder does even less than virt-sysprep. In particular it doesn't touch /.autorelabel. However the prepared templates that we ship with virt-builder were all virt-sysprepped before being uploaded, and so all have /.autorelabel in them. What files virt-sysprep creates or modifies ------------------------------------------- This is my analysis of what files virt-sysprep creates or modifies: firstboot: - creates scripts and more (brand new files) - shared with virt-builder reconfiguration: - creates /.unconfigured (brand new) hostname: - creates various files depending on distro (eg /etc/hostname) - files probably exist already, but might not - shared with virt-builder machine_id: - creates empty /etc/machine-id - probably this file exists already net_hostname: - modifies /etc/sysconfig/network-scripts/ifcfg-* net_hwaddr: - modifies /etc/sysconfig/network-scripts/ifcfg-* pacct_log: - touches /var/account/pacct or /var/log/account/pacct password: - edits /etc/shadow directly - shared with virt-builder random_seed: - creates seed file - seed file may or may not exist before - shared with virt-builder timezone: - creates/updates /etc/localtime symlink - file almost certainly exists before, but might be a regular file - shared with virt-builder user_account: - uses Augeas to edit /etc/passwd, /etc/shadow, /etc/group - Augeas does not preserve or set SELinux labels [in this case] What files virt-builder creates or modifies ------------------------------------------- edit option: - edits a file - copies xattrs from old file to new file (thanks Pino!) link, mkdir, upload, write options: - creates new links/directories/files under user control - currently no way to specify the SELinux label of the new file - would user know how to specify it? install, update options: - complicated: see longer section below Plus, shared modules with virt-sysprep, see notes above: - firstboot - hostname - password - random_seed - timezone Virt-builder --install and --update options ------------------------------------------- These options allow you to install new packages, or update existing packages. They run 'yum' in the context of the libguestfs appliance. The appliance normally has SELinux disabled and no policy loaded. It is plausible that RPM packages could contain the right SELinux labels, and that yum could be setting labels correctly (by setting the security.selinux xattr) even though SELinux is disabled. However I could not say for sure, so I did the following test instead: $ virt-builder fedora-20 --install /usr/sbin/tcpdump $ guestfish --ro -a fedora-20.img -i ><fs> getxattrs /usr/sbin/tcpdump [no output] It seems from this that RPM is not setting labels, instead it relies on restorecond/etc to do the labelling of files that it creates. Conclusion ---------- I don't think there is any plausible way we can make --install or --update work without requiring a full filesystem relabel on first boot. However I think we can try harder to copy attributes from old files to new files, which will make many cases above work (especially for virt-sysprep which seems tractable). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org

2 4

[PATCH] builder: proper consider subkeys in index files
by Pino Toscano 21 Jan '14

21 Jan '14

The index files already allowed the 'key[subkey]=...' syntax for keys, but considering such string as whole key. Proper split the parsing and the handling of the subkeys, so they can be searched a bit easier. This causes no actual behaviour changes, it is just internal micro-refactoring. --- builder/index-parser-c.c | 8 +++++--- builder/index-scan.l | 9 ++++++++- builder/index-struct.c | 1 + builder/index-struct.h | 1 + builder/index_parser.ml | 44 ++++++++++++++++++++++++++------------------ 5 files changed, 41 insertions(+), 22 deletions(-) diff --git a/builder/index-parser-c.c b/builder/index-parser-c.c index 17e680b..baf2b62 100644 --- a/builder/index-parser-c.c +++ b/builder/index-parser-c.c @@ -83,11 +83,13 @@ virt_builder_parse_index (value filenamev) for (j = 0, fields = sections->fields; fields != NULL; j++, fields = fields->next) { - v = caml_alloc_tuple (2); + v = caml_alloc_tuple (3); sv = caml_copy_string (fields->key); - Store_field (v, 0, sv); /* (key, value) */ - sv = caml_copy_string (fields->value); + Store_field (v, 0, sv); /* (key, subkey, value) */ + sv = caml_copy_string (fields->subkey ? fields->subkey : ""); Store_field (v, 1, sv); + sv = caml_copy_string (fields->value); + Store_field (v, 2, sv); Store_field (fv, j, v); /* assign to return array of fields */ } diff --git a/builder/index-scan.l b/builder/index-scan.l index 9a6a0e3..7a9618f 100644 --- a/builder/index-scan.l +++ b/builder/index-scan.l @@ -58,10 +58,17 @@ extern void yyerror (const char *); /* field=value or field[subfield]=value */ ^[A-Za-z0-9_.]+("["[A-Za-z0-9_,.]+"]")?"=".*\n { - size_t i = strcspn (yytext, "="); + size_t i = strcspn (yytext, "=["); yylval.field = malloc (sizeof (struct field)); yylval.field->next = NULL; yylval.field->key = strndup (yytext, i); + if (yytext[i] == '[') { + size_t j = strcspn (yytext+i+1, "]"); + yylval.field->subkey = strndup (yytext+i+1, j); + i += 1+j+2; + } else { + yylval.field->subkey = NULL; + } /* Note we chop the final \n off here. */ yylval.field->value = strndup (yytext+i+1, yyleng-(i+2)); return FIELD; diff --git a/builder/index-struct.c b/builder/index-struct.c index 26bed24..fe5b0e3 100644 --- a/builder/index-struct.c +++ b/builder/index-struct.c @@ -52,6 +52,7 @@ free_field (struct field *field) if (field) { free_field (field->next); free (field->key); + free (field->subkey); free (field->value); free (field); } diff --git a/builder/index-struct.h b/builder/index-struct.h index ac8a3dd..f92e01d 100644 --- a/builder/index-struct.h +++ b/builder/index-struct.h @@ -32,6 +32,7 @@ struct section { struct field { struct field *next; char *key; + char *subkey; char *value; }; diff --git a/builder/index_parser.ml b/builder/index_parser.ml index 453a3a1..4b070cb 100644 --- a/builder/index_parser.ml +++ b/builder/index_parser.ml @@ -101,7 +101,7 @@ let print_entry chan (name, { printable_name = printable_name; type sections = section array and section = string * fields (* [name] + fields *) and fields = field array -and field = string * string (* key + value *) +and field = string * string * string (* key + subkey + value *) (* Calls yyparse in the C code. *) external parse_index : string -> sections = "virt_builder_parse_index" @@ -149,12 +149,13 @@ let get_index ~prog ~debug ~downloader ~sigchecker source = fun (n, fields) -> let fseen = Hashtbl.create 13 in List.iter ( - fun (field, _) -> - if Hashtbl.mem fseen field then ( + fun (field, subkey, _) -> + let hashkey = (field, subkey) in + if Hashtbl.mem fseen hashkey then ( eprintf (f_"virt-builder: index is corrupt: %s: field '%s' appears two or more times\n") n field; corrupt_file () ); - Hashtbl.add fseen field true + Hashtbl.add fseen hashkey true ) fields ) sections; @@ -162,25 +163,32 @@ let get_index ~prog ~debug ~downloader ~sigchecker source = let entries = List.map ( fun (n, fields) -> + let find_elem key subkey fields = + match List.filter ( + fun (iterkey, itersubkey, itervalue) -> + iterkey = key && itersubkey = subkey + ) fields with + | [] -> raise Not_found + | (_, _, value) :: _ -> value in let printable_name = - try Some (List.assoc "name" fields) with Not_found -> None in + try Some (find_elem "name" "" fields) with Not_found -> None in let osinfo = - try Some (List.assoc "osinfo" fields) with Not_found -> None in + try Some (find_elem "osinfo" "" fields) with Not_found -> None in let file_uri = - try make_absolute_uri (List.assoc "file" fields) + try make_absolute_uri (find_elem "file" "" fields) with Not_found -> eprintf (f_"virt-builder: no 'file' (URI) entry for '%s'\n") n; corrupt_file () in let signature_uri = - try Some (make_absolute_uri (List.assoc "sig" fields)) + try Some (make_absolute_uri (find_elem "sig" "" fields)) with Not_found -> None in let checksum_sha512 = - try Some (List.assoc "checksum[sha512]" fields) + try Some (find_elem "checksum" "sha512" fields) with Not_found -> - try Some (List.assoc "checksum" fields) + try Some (find_elem "checksum" "" fields) with Not_found -> None in let revision = - try int_of_string (List.assoc "revision" fields) + try int_of_string (find_elem "revision" "" fields) with | Not_found -> 1 | Failure "int_of_string" -> @@ -188,9 +196,9 @@ let get_index ~prog ~debug ~downloader ~sigchecker source = n; corrupt_file () in let format = - try Some (List.assoc "format" fields) with Not_found -> None in + try Some (find_elem "format" "" fields) with Not_found -> None in let size = - try Int64.of_string (List.assoc "size" fields) + try Int64.of_string (find_elem "size" "" fields) with | Not_found -> eprintf (f_"virt-builder: no 'size' field for '%s'\n") n; @@ -200,7 +208,7 @@ let get_index ~prog ~debug ~downloader ~sigchecker source = n; corrupt_file () in let compressed_size = - try Some (Int64.of_string (List.assoc "compressed_size" fields)) + try Some (Int64.of_string (find_elem "compressed_size" "" fields)) with | Not_found -> None @@ -209,13 +217,13 @@ let get_index ~prog ~debug ~downloader ~sigchecker source = n; corrupt_file () in let expand = - try Some (List.assoc "expand" fields) with Not_found -> None in + try Some (find_elem "expand" "" fields) with Not_found -> None in let lvexpand = - try Some (List.assoc "lvexpand" fields) with Not_found -> None in + try Some (find_elem "lvexpand" "" fields) with Not_found -> None in let notes = - try Some (List.assoc "notes" fields) with Not_found -> None in + try Some (find_elem "notes" "" fields) with Not_found -> None in let hidden = - try bool_of_string (List.assoc "hidden" fields) + try bool_of_string (find_elem "hidden" "" fields) with | Not_found -> false | Failure "bool_of_string" -> -- 1.8.3.1

2 5

[PATCH] builder: allow more empty lines in index files
by Pino Toscano 21 Jan '14

21 Jan '14

Improve the index grammar tp allow more than one empty line between sections, and to allow any number of empty lines at the end of the file (after the last section). --- builder/index-parse.y | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/builder/index-parse.y b/builder/index-parse.y index f5e551b..a8d2f62 100644 --- a/builder/index-parse.y +++ b/builder/index-parse.y @@ -80,10 +80,10 @@ index: { parsed_index = $2; } sections: - section + section emptylines { $$ = $1; } - | section EMPTY_LINE sections - { $$ = $1; $$->next = $3; } + | section EMPTY_LINE emptylines sections + { $$ = $1; $$->next = $4; } section: SECTION_HEADER fields @@ -113,6 +113,12 @@ continuations: free ($1); free ($2); } +emptylines: + /* empty */ + {} + | EMPTY_LINE emptylines + {} + %% void -- 1.8.3.1

2 1

[PATCH 1/2] sysprep: Update comments.
by Richard W.M. Jones 21 Jan '14

21 Jan '14

--- sysprep/sysprep_operation.mli | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sysprep/sysprep_operation.mli b/sysprep/sysprep_operation.mli index 61dde72..eb89db4 100644 --- a/sysprep/sysprep_operation.mli +++ b/sysprep/sysprep_operation.mli @@ -16,14 +16,16 @@ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. *) -(** Structure used to describe sysprep operations. *) +(** Defines the interface between the main program and sysprep operations. *) val prog : string type flag = [ `Created_files ] type callback = Guestfs.guestfs -> string -> flag list +(** [callback g root] is called to do work. *) +(** Structure used to describe sysprep operations. *) type operation = { name : string; (** Operation name, also used to enable the operation on the command -- 1.8.4.2

1 1

Libguestfs can't launch with one of the disk images in the RHEV cluster
by Исаев Виталий Анатольевич 21 Jan '14

21 Jan '14

Dear Libguestfs developers, I faced with an unexpected behavior of the quite old version of libguestfs (1.16.34) running on the RHEV 3.1 hypervisor which is part of the RHEV cluster with SCSI storage. My issue is described in the attachment. Unfortunately, updating is not suitable for the project I'm working on due to the licensing and security reasons. I am going to send a request to the RH Customer Support, but I prefer to ask the community first. Thank you in advance. Vitaly Isaev Software engineer Information security department Fintech JSC, Moscow, Russia

3 6

LVM mounting issue
by Исаев Виталий Анатольевич 21 Jan '14

21 Jan '14

Hello, It's me again with one more misunderstanding of libguestfs usage... Guestfish (as well as python libguestfs module) refuse to mount logical volume containing the root partition of the VM's filesystem. According to this short guide<http://rwmj.wordpress.com/2009/04/07/libguestfs-lvm-support/> I am trying to mount /dev/vg_kojit/lv_root as "/", but nothing works: [root@rhevh1 integrity]# [root@rhevh1 integrity]# [root@rhevh1 integrity]# [root@rhevh1 integrity]# guestfish Welcome to guestfish, the libguestfs filesystem interactive shell for editing virtual machine filesystems. Type: 'help' for help on commands 'man' to read the manual 'quit' to quit the shell ><fs> add /dev/dm-40 ><fs> run ><fs> inspect_os ><fs> pvs /dev/vda2 unknown device ><fs> lvs /dev/vg_kojit/lv_root /dev/vg_kojit/lv_swap ><fs> mount /dev/vg_kojit/lv_root / libguestfs: error: mount: mount_stub: /dev/vg_kojit/lv_root: No such file or directory ><fs> ><fs> The actual information about VM's block devices can be seen on the screenshot attached to this message. I would be grateful for your help Sincerely, Vitaly Isaev Software engineer Information security department Fintech JSC, Moscow, Russia

2 7

newlines with write-append
by Olaf Hering 20 Jan '14

20 Jan '14

Silly bash scripts have stuff like below to get things done, but equally silly guestfish scripts fail to add the required newline. Why is that? echo "$dev1 $mnt1 $fs $opts 1 2" >> /etc/fstab echo "$dev2 $mnt2 $fs $opts 1 2" >> /etc/fstab write-append /etc/fstab "$dev1 $mnt1 $fs $opts 1 2" : \ write-append /etc/fstab "$dev2 $mnt2 $fs $opts 1 2" : \ Even adding variants of \n does not help. Simple testcase: write /etc/fstab "#\n" : \ write-append /etc/fstab '#1\n' : \ write-append /etc/fstab '#2\\n' : \ cat /etc/fstab : \ quit ... libguestfs: trace: cat = "#\n#1\n#2\\n" #\n#1\n#2\\n libguestfs: trace: shutdown ... I guess using "sh" will be my workaround. Olaf

2 2