Fwd: Guestfish Ansible Modules using Python Bindings
by Richard W.M. Jones
----- Forwarded message from Petros Petrou <ppetrou(a)redhat.com> -----
Date: Mon, 20 Apr 2020 09:29:37 +0100
From: Petros Petrou
To: Richard Jones
Subject: Guestfish Ansible Modules using Python Bindings
Hi Richard,
I have been experimenting with guestfish and rhel qcow2 cloud images
the last few months.
I was challenged in a recent assignment on how to customize the RHEL 8
qcow image using cloud libraries to meet complex setups suchs a
multiple partitions, lvm storage, openscap compliance etc as the
default image is a single standard partition which is not suitable for
production.
Although the customer moved forward with kickstart to build their images I
decided to put some effort on this and developed a python module to wrap
the libguestfish python bindings and also developed a few ansible modules
to support partitions, filesystems and lvm storage so as to see how much
and how easy I can customize a cloud image using guestfish.
I did some research beforehand and did not find much info on the web other
than your website and blog so I assume I am not reinventing the wheel here
:)
My coding is in my github and will soon write a blog and try to present to
the Red Hat Automation CoP and also try to push the Ansible Modules to the
community.
If you have any views or suggestions on this please let me know. I have
found guestfish and supermin a very interesting library to work with and
learned a lot.
https://github.com/ppetrou/infra_server_kvm/tree/master/roles/infra_serve...
https://github.com/ppetrou/infra_server_kvm/blob/master/roles/infra_serve...
https://github.com/ppetrou/infra_server_kvm/tree/master/roles/infra_serve...
Thank you,
Petros
----- End forwarded message -----
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
4 years, 1 month
[PATCH] python: Fix UnicodeError in inspect_list_applications2() (RHBZ#1684004)
by Sam Eiderman
The python3 bindings create unicode objects from application strings
on the guest (i.e. installed rpm, deb packages).
It is documented that rpm package fields such as description should be
utf8 encoded - however in some cases they are not a valid unicode
string, on SLES11 SP4 the following packages fail to be converted to
unicode using guestfs_int_py_fromstring() (which invokes
PyUnicode_FromString()):
PackageKit
aaa_base
coreutils
dejavu
desktop-data-SLED
gnome-utils
hunspell
hunspell-32bit
hunspell-tools
libblocxx6
libexif
libgphoto2
libgtksourceview-2_0-0
libmpfr1
libopensc2
libopensc2-32bit
liborc-0_4-0
libpackagekit-glib10
libpixman-1-0
libpixman-1-0-32bit
libpoppler-glib4
libpoppler5
libsensors3
libtelepathy-glib0
m4
opensc
opensc-32bit
permissions
pinentry
poppler-tools
python-gtksourceview
splashy
syslog-ng
tar
tightvnc
xorg-x11
xorg-x11-xauth
yast2-mouse
This is a surgical fix for inspect_list_applications2()'s description
field.
Signed-off-by: Sam Eiderman <sameid(a)google.com>
---
generator/python.ml | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/generator/python.ml b/generator/python.ml
index f0d6b5d96..7394a943a 100644
--- a/generator/python.ml
+++ b/generator/python.ml
@@ -170,6 +170,14 @@ and generate_python_structs () =
function
| name, FString ->
pr " value = guestfs_int_py_fromstring (%s->%s);\n" typ name;
+ (match typ, name with
+ | "application", "app_description"
+ | "application2", "app2_description" ->
+ pr " if (value == NULL) {\n";
+ pr " value = guestfs_int_py_fromstring (\"\");\n";
+ pr " PyErr_Clear ();\n";
+ pr " }\n";
+ | _ -> pr ""; );
pr " if (value == NULL)\n";
pr " goto err;\n";
pr " PyDict_SetItemString (dict, \"%s\", value);\n" name;
--
2.26.1.301.g55bc3eb7cb9-goog
4 years, 1 month
ANNOUNCE: virt-v2v 1.42 - a tool for importing VMs from VMware (and other foreign hypervisors) to KVM
by Richard W.M. Jones
Virt-v2v is a program for converting virtual machines to run on KVM,
ones imported from other hypervisors like VMware, Xen and more, and
with the companion tool virt-p2v from physical machines too.
I'm pleased to announce the release of virt-v2v 1.42. This is a new
stable branch. Significantly the tool is now released separately and
on a different schedule from libguestfs.
Manual: http://libguestfs.org/virt-v2v.1.html
Download: http://download.libguestfs.org/virt-v2v/1.42-stable/
Git repo: https://github.com/libguestfs/virt-v2v
Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=9181
Release notes (also available on the web:
http://libguestfs.org/virt-v2v-release-notes-1.42.1.html )
These are the release notes for virt-v2v 1.42, released on
16th April 2020.
New features
This is the first release where virt-v2v lives in a separate repository
from libguestfs. The two projects are now broadly decoupled from one
another.
Add a new -o json output mode. Primarily this is used to do
conversions to KubeVirt (Pino Toscano).
Use new libvirt "<firmware>" feature to get the source guest firmware
(Pino Toscano).
virt-v2v -o rhv-upload new option -oo rhv-disk-uuid allows disk UUIDs
to be specified. Also -oo rhv-cafile is now optional (Pino Toscano).
Conversions over ssh now use nbdkit-ssh-plugin(1) instead of the QEMU
ssh driver. Similarly nbdkit-curl-plugin(1) is used instead of the
QEMU curl driver. This allowed us to add more flexible features such
as password authentication, bandwidth throttling (new --bandwidth
option), readahead, and automatic retry on network failures.
For Windows guests, QEMU Guest Agent MSI may now be installed (Tomáš
Golembiovský).
Other fixes
In -o libvirt mode, support Windows Server 2019 (Pino Toscano).
Fix Ubuntu Server conversions (Pino Toscano).
Fix installation of qemu-ga by only installing arch-specific files in
the guest (Pino Toscano).
Delay installation of qemu-ga until after virtio-win drivers have been
installed and rebooted (Tomáš Golembiovský).
Save the log from running RHEV-APT installer to allow debugging (Tomáš
Golembiovský).
Check RHV cluster exists before trying to convert when using -o rhv-
upload mode. Also allows us to detect and reject various other
conditions early. (Pino Toscano).
Label nbdkit sockets correctly for SELinux/sVirt (Martin Kletzander).
You can use a block device as the Windows virtio driver ISO.
Multiple fixes to RHV uploads: Set "DISKTYPE" field correctly when
converting to RHV and VDSM. Properly clean up on failure. Display
disk ID in error messages, and log script parameters, to help with
debugging. Multiple code cleanups. (Nir Soffer).
Support conversions to RHV in qcow2 format (Nir Soffer).
Fix detection of disk status and failures after conversion to RHV
(Daniel Erez).
Cancel disk transfer and remove uploaded disks on failure of conversion
to RHV (Pino Toscano).
Images containing small holes (sparse regions) should now convert
faster (Nir Soffer).
The nbdkit-cacheextents-filter(1) is used to accelerate sources which
have slow sparseness detection, primarily this means VMware sources
using VDDK (Martin Kletzander).
Require at least 100 free inodes on each guest filesystem before doing
conversion, since lack of inodes could cause conversion failures (Pino
Toscano).
Fix osinfo output for CentOS 8 conversions (Pino Toscano).
VMware tools are now removed from Windows guests automatically in most
cases (Pino Toscano).
-i ova mode no longer reads the whole input OVF into memory, but parses
it off disk (Pino Toscano).
When converting to OpenStack, we now wait up to 5 minutes (instead of
60 seconds) for the Cinder volume to get attached to the conversion
appliance, since in some cases it was taking a long time.
If using a new enough version of nbdkit, virt-v2v logs should be much
less verbose.
Security
There were no security-related bugs found in this release.
Build changes
Libguestfs ≥ 1.40 is required, it is now packaged and distributed
separately from virt-v2v. For developers you can use a locally built
(and not installed) copy of libguestfs.
Libvirt is now required to build virt-v2v. Additionally you will
require the OCaml bindings to libvirt (https://libvirt.org/ocaml/),
although a copy is bundled for now (it will be removed later). (Pino
Toscano).
Libosinfo is required to build virt-v2v. It is used to query
information about guest drivers. (Pino Toscano).
Fixes for OCaml ≥ 4.10. The minimum version of OCaml required is still
4.01, but may be increased to 4.05 in future.
Test conversions of Debian 9 and Fedora 29 (Pino Toscano).
Various fixes to srcdir != builddir (Pino Toscano).
Remove gnulib modules not used by virt-v2v (Pino Toscano).
Internals
When converting SUSE guests, always try to install QXL driver (Mike
Latimer).
Two new modules (Nbdkit and Nbdkit_sources) factor out generic nbdkit
operations and nbdkit source operations respectively.
Format Python code to comply with PEP 8 / pycodestyle (Pino Toscano).
Tests have been moved to the tests/ subdirectory, and manuals to the
docs/ subdirectory.
Bugs fixed
https://bugzilla.redhat.com/1791802
virt-v2v does not install qemu-ga on EL8 guest
https://bugzilla.redhat.com/1791257
update-crypto-policies command example is incorrect in
virt-v2v-input-xen
https://bugzilla.redhat.com/1791240
[RFE] Make the rhv-cafile optional
https://bugzilla.redhat.com/1785528
Should remove info about "Remove VMware tools from Windows guests"
in virt-v2v-input-vmware man page
https://bugzilla.redhat.com/1746699
Can't import guest from export domain to data domain on rhv4.3 due
to error "Invalid parameter: 'DiskType=1'"
https://bugzilla.redhat.com/1733168
virt-v2v: Use scp -T in -i vmx -it ssh mode
https://bugzilla.redhat.com/1723305
Delete info "export PATH=/path/to/nbdkit-1.1.x:$PATH" in
virt-v2v-input-vmware manual page
https://bugzilla.redhat.com/1691659
virt-v2v should show a message when qemu-guest-agent is installed
in guest successfully during conversion
https://bugzilla.redhat.com/1690574
virt-v2v fails to import a guest while cannot find
`file_architecture` for a file
https://bugzilla.redhat.com/1680361
[v2v][RHV][Scale] v2v Migration to RHV failed on timed out waiting
for transfer to finalize
https://bugzilla.redhat.com/1626503
Unable to maintain static IP address configuration post VM
migration
https://bugzilla.redhat.com/1612653
Guest has no disk after rhv-upload converting if target data domain
has similar name with other data domain on rhv4.2
https://bugzilla.redhat.com/1605242
Update nbdkit info for vddk in v2v man page
https://bugzilla.redhat.com/1584678
On W2K12r2 rhev-apt does not run non-interactively, causing race
when starting rhev-apt service from the command line
https://bugzilla.redhat.com/1518539
Macvtap network will be lost during v2v conversion
SEE ALSO
virt-v2v(1).
Previous release notes covering virt-v2v can be found in the libguestfs
project: guestfs-release-notes-1.40(1).
AUTHORS
Daniel Erez
Richard W.M. Jones
Tomáš Golembiovský
Martin Kletzander
Mike Latimer
Nir Soffer
Pino Toscano
Ming Xie
COPYRIGHT
Copyright (C) 2009-2020 Red Hat Inc.
LICENSE
This program is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation; either version 2 of the License, or (at your
option) any later version.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
4 years, 1 month
[PATCH nbdkit 0/9] Generic vector, and pass $nbdkit_stdio_safe to shell scripts.
by Richard W.M. Jones
This was a rather longer trip around the houses than I anticipated!
The basic purpose of the patch series is to set $nbdkit_stdio_safe to
"0" or "1" in sh and eval plugin scripts.
To do that, I ended up adding a nicer way to manipulate environ lists,
and to do that, I ended up adding a whole generic vector
implementation which is applicable in a lot of different places.
Rich.
4 years, 1 month
[PATCH nbdkit UNFINISHED] Add the ability to write plugins in golang.
by Richard W.M. Jones
Sorry Dan, but I really do dislike golang with a passion :-)
Here is a patch that allows you to write nbdkit plugins in golang. As
with C, OCaml and Rust, you can write a plugin in Go which compiles
directly to a .so file that can be loaded into golang, so in that
sense it works completely differently from scripting language plugins
like Perl and Python where there's an nbdkit-<lang>-plugin that
intermediates between nbdkit and the user’s code.
With that said, there are many problems. The root cause of most of
them is that you cannot pass Go pointers to C, perhaps because the
golang developers never heard of registering extra GC roots[1], even
though that is common in non-Blub languages like OCaml and Haskell.
This leads to awkward complications nicely summarised in this page:
https://eli.thegreenplace.net/2019/passing-callbacks-and-pointers-to-cgo/
This requires that every callback has a wrapper across 3 files. Oh
and these wrappers cannot be in a library module, they must be copied
into the plugin source code, AND they must be commented out by the
end-user by hand if the callback is not called. (There may be a way
around the latter issues, but I could not work out how.) This means
plugins have tons of duplicated source code.
Other issues that are not related to that one:
- We have to link the nbdkit module with
-Wl,--unresolved-symbols=ignore-in-object-files because of the way
nbdkit plugins have deliberately unresolved symbols. See the
libnbdkit.so proposal on the mailing list for another way to solve
this.
- Be nice if ./configure could check that golang >= 1.5 since that
was the first version that introduced shared libraries.
- Since initialization is not synchronous in golang, you cannot rely
on anything being initialized in the plugin before nbdkit starts
calling in. For this reason I worked around it by having
plugin_init() call a start up function (func init_plugin()) where
all golang initialization must be done. Otherwise:
nbdkit: golang plugin:
dlopen ("plugin.so");
let's start initializing
init = dlsym ("plugin_init");
return &plugin (uninitialized!)
plugin = init ();
plugin.load ()
hey, I'm still initializing!
- Related to the previous point: Be nice to move plugin_init() into
the nbdkit module. However I don't believe this is possible
because this function has to call into the plugin (in main module).
- Tests are a joke at the moment. We would really need a test which
properly exercises threads / parallel client connections, so we can
be sure that the nbdkit thread & golang goroutine models do not
conflict in some way. (I don't think they do, but need to check).
- Current test func pluginPRead() needs to be completed.
- Documentation needs fixing. I didn't want to write too many docs
until I know finally how plugins would work.
Rich.
[1] Reading the proposal here confirms my suspicions, since there is a
much simpler, better and more obvious solution than what is proposed:
https://github.com/golang/proposal/blob/master/design/12416-cgo-pointers.md
4 years, 1 month
[nbdkit PATCH v2 0/3] more consistent stdin/out handling
by Eric Blake
In v2:
- use int instead of bool in the public header
- split the tests from the code
- don't overload test-layers; instead, add new tests
- add a missing fflush exposed by the new tests
- other minor cleanups
Eric Blake (3):
server: Add nbdkit_stdio_safe
server: Sanitize stdin/out before running plugin code
server: More tests of stdin/out handling
docs/nbdkit-plugin.pod | 23 ++++-
plugins/sh/nbdkit-sh-plugin.pod | 4 +-
include/nbdkit-common.h | 1 +
tests/Makefile.am | 23 +++++
server/internal.h | 2 +
server/background.c | 14 +--
server/captive.c | 10 +-
server/connections.c | 12 ---
server/main.c | 38 ++++++-
server/nbdkit.syms | 1 +
server/public.c | 18 +++-
server/test-public.c | 23 ++++-
plugins/sh/sh.c | 7 +-
tests/test-single-sh.sh | 78 +++++++++++++++
tests/test-stdio.sh | 95 ++++++++++++++++++
tests/test-stdio-plugin.c | 170 ++++++++++++++++++++++++++++++++
16 files changed, 489 insertions(+), 30 deletions(-)
create mode 100755 tests/test-single-sh.sh
create mode 100755 tests/test-stdio.sh
create mode 100644 tests/test-stdio-plugin.c
--
2.26.0
4 years, 1 month
[PATCH nbdkit PRELIMINARY] file: Move file operators to a new fileops mini-library
by Richard W.M. Jones
There's a lot of code in nbdkit-file-plugin which it would be nice to
reuse elsewhere. One possible approach (as outlined here) is simply
to move the file callbacks (like file.pread, file.pwrite, file.zero
etc) to a new mini-library. They can then be consumed by other
plugins fairly easily by doing:
static void *
foo_open (int readonly)
{
struct fileops *fops;
int fd, flags;
/* Allocate *fops */
/* Set up fd however you want */
if (init_fileops (fd, fops) == -1) {
free (fops);
return NULL;
}
return fops;
}
static struct nbdkit_plugin plugin = {
.name = "foo",
.open = foo_open,
.close = foo_close,
FILEOPS_CALLBACKS
};
If we did this then it would only work for plugins which need to serve
exactly one whole file or block device. In fact the only plugins we
could possibly use it for are iso and tmpdisk.
We can't use it for plugins like split, partitioning, linuxdisk
because those don't only serve a single file. Instead they all either
serve multiple files or have some extra structure (like a virtual
partition table) around a single file.
Another approach (not explored yet) might be to expose more
fundamental primitives like "zeroing part of a file efficiently".
I think possibly best is a mixed approach: We add fileops, but also we
expose fundamental primitives (in libutils). Simpler plugins would
use fileops, others would have a more complex implementation using the
fundamental primitives.
Rich.
4 years, 1 month