-----Original Message----- From: Richard W.M. Jones [mailto:rjones@redhat.com] Sent: Tuesday, January 14, 2014 4:42 PM To: Исаев Виталий Анатольевич Cc: libguestfs(a)redhat.com Subject: Re: [Libguestfs] Libguestfs can't launch with one of the disk images in the RHEV cluster On Tue, Jan 14, 2014 at 08:07:43AM +0000, Исаев Виталий Анатольевич wrote: libguestfs runs qemu with the command line above. qemu tries to open the /dev/mapper/1a9... file. qemu reports that it cannot open that file. Unfortunately qemu's error messages are very poor. However there are a few possibilities: (a) An actual permissions issue. Since you seem to be running this as root, this doesn't seem to be likely, but you should check that anyway. Are there SELinux AVCs? (b) qemu cannot open the backing file. Try running: qemu-img info /dev/mapper/1a9... and if it has a backing file, check that the backing file(s) [recursively] can be opened too. (c) Also check that the backing file paths are not relative. If they are you will need to run your script from the correct directory so that the relative paths are accessible. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v Dear Rich, thank you for a prompt reply on my question. The similar problems have been found with all of the rest Thin Provisioned disks in the cluster, while all the Preallocated disks were handled with libguestfs correctly. I guess these issues were caused by (b) and probably (c) reasons: The backing file of any of the thin provisioned disks does not exist. For instance let’s consider the /dev/mapper/1a9aa971--f81f--4ad8--932f--607034c924fc-666faa62--da73--4465--aed2--912119fcf67f symbolic link pointing to /dev/dm-30 : [root@rhevh1 mapper]# pwd /dev/mapper [root@rhevh1 mapper]# qemu-img info 1a9aa971--f81f--4ad8--932f--607034c924fc-666faa62--da73--4465--aed2--912119fcf67f image: 1a9aa971--f81f--4ad8--932f--607034c924fc-666faa62--da73--4465--aed2--912119fcf67f file format: qcow2 virtual size: 40G (42949672960 bytes) disk size: 0 cluster_size: 65536 backing file: ../6439863f-2d4e-48ae-a150-f9054650789c/cbe36298-6397-4ffa-ba8c-5f64e90023e5 [root@rhevh1 mapper]# ll ../6439863f-2d4e-48ae-a150-f9054650789c/cbe36298-6397-4ffa-ba8c-5f64e90023e5 ls: cannot access ../6439863f-2d4e-48ae-a150-f9054650789c/cbe36298-6397-4ffa-ba8c-5f64e90023e5: No such file or directory Note that /dev/dm-30 is not accessible with libguestfs. Now I am trying to find the files with the same name. As a result I receive three symbolic links pointing to /dev/1a9aa971-f81f-4ad8-932f-607034c924fc/cbe36298-6397-4ffa-ba8c-5f64e90023e5: [root@rhevh1 mapper]# find / -name cbe36298-6397-4ffa-ba8c-5f64e90023e5 /dev/1a9aa971-f81f-4ad8-932f-607034c924fc/cbe36298-6397-4ffa-ba8c-5f64e90023e5 /var/lib/stateless/writable/rhev/data-center/mnt/blockSD/1a9aa971-f81f-4ad8-932f-607034c924fc/images/6439863f-2d4e-48ae-a150-f9054650789c/cbe36298-6397-4ffa-ba8c-5f64e90023e5 /rhev/data-center/mnt/blockSD/1a9aa971-f81f-4ad8-932f-607034c924fc/images/6439863f-2d4e-48ae-a150-f9054650789c/cbe36298-6397-4ffa-ba8c-5f64e90023e5 In turn, the /dev/1a9aa971-f81f-4ad8-932f-607034c924fc/cbe36298-6397-4ffa-ba8c-5f64e90023e5 file is a symbolic link which points to the /dev/dm-19 At last I am trying to launch libguestfs with block device directly: [root@rhevh1 mapper]# qemu-img info /dev/dm-19 image: /dev/dm-19 file format: raw virtual size: 40G (42949672960 bytes) disk size: 0 [root@rhevh1 mapper]# python Python 2.6.6 (r266:84292, Oct 12 2012, 14:23:48) [GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. [] [] ['/dev/vda1', '/dev/vda2'] ['/dev/vda1'] Now I’m a little bit confused with the results of my research. I found that VM with the only disk attached has at least two block devices mapped to the hypervisor’s file system in fact – I mean /dev/dm-19 (raw) and /dev/dm-30 (qcow2). The RHEV-M API (aka Python oVirt SDK) provides no info about the first one, but the second cannot be accessed from libguestfs. I have an urgent need to work with a chosen VM disk images through the libguestfs layer, but I don’t know which images belong to every VM exactly. It seems like I’m going the hard way :) Sincerely, Vitaly Isaev Software engineer Information security department Fintech JSC, Moscow, Russia

-----Original Message----- From: Richard W.M. Jones [mailto:rjones@redhat.com] Sent: Tuesday, January 14, 2014 9:43 PM To: Исаев Виталий Анатольевич Cc: libguestfs(a)redhat.com Subject: Re: [Libguestfs] Libguestfs can't launch with one of the disk images in the RHEV cluster On Tue, Jan 14, 2014 at 02:57:35PM +0000, Исаев Виталий Анатольевич wrote: This works because you're accessing the backing disk, not the top disk. Since the backing disk (in this case) doesn't itself have a backing disk, qemu has no problem opening it. Basically you need to find out which directory RHEV-M itself starts qemu in. Try going onto the node and doing: ps ax | grep qemu ls -l /proc/PID/cwd substituting PID for some of the qemu process IDs. My guess would be some subdirectory of /rhev/data-center/mnt/blockSD/ Then start your test script from that directory. Another thing you could do is to file a bug against oVirt asking them not to use relative paths for backing disks, since plenty of people have problems with this. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org Thank you, Richard. I’ve posted a message to the RH bugtracker: https://bugzilla.redhat.com/show_bug.cgi?id=1053684 Further work on this problem made the things even more complicated, because I found that several qcow2 disks have qcow2 disks as backing disks in turn. So now I have to resolve qcow2 to raw disks recursively in order to access them with libguestfs. Vitaly Isaev Software engineer Information security department Fintech JSC, Moscow, Russia

-----Original Message----- From: Federico Simoncelli [mailto:fsimonce@redhat.com] Sent: Friday, January 17, 2014 9:04 PM To: libguestfs(a)redhat.com Cc: Исаев Виталий Анатольевич Subject: Re: [Libguestfs] Libguestfs can't launch with one of the disk images in the RHEV cluster ----- Original Message ----- Yes, the full path to the images is: /rhev/data-center/mnt/blockSD/<sdUUID>/images/<imgUUID>/ sdUUID is the uuid of the storage domain (vg name) where the images are stored and imgUUID is the uuid of the image (as reported in the "Disks" tab in the webadmin). The symlinks and the LVs are managed by vdsm (activated when the VM starts, deactivated as soon as the VM is stopped). If you just want to access these images once and you're sure that they are not in use you could activate the relevant LVs and create the symlinks yourself on any machine (no need of oVirt/vdsm), e.g. /tmp/ovirt-images/<imgUUID>/<volUUID1> -> /dev/dm-xx /tmp/ovirt-images/<imgUUID>/<volUUID2> -> /dev/dm-xx /tmp/ovirt-images/<imgUUID>/<volUUID3> -> /dev/dm-xx this could even be automated eventually (from the qemu-img output). If instead you're trying to integrate with oVirt and you need a more reliable/automated solution I invite you start a thread on the oVirt mailing list. The imgUUIDs can be found also with: lvs -o +tags looking for the tags starting with "IU_". -- Federico Hello, Federico, thank you for your reply and invitation. I'm working on the project of RHEV enforcement. I’m trying to develop a program that will monitor the integrity of the files stored on the virtual machine disk images. There are several conditions for the RHEV environment: 1. We have a fixed number of the VMs that will not change after the RHEV HA cluster deployment; 2. VM’s key system directories [‘/bin’, ’/sbin’, ’/lib’, ’boot’ etc.] content must not change after cluster deployment; 3. Checksums of the files from system directories must be compared to the initial checksums (computed after the cluster deployment) every time when VM is starting; So what I am trying to do is just: 1. To define which disk belongs to every VM; //python oVirt SDK 2. To write deployment script that will compute the initial checksums of the VMs; //libguestfs 3. To store these sums on every RHEV-H (ovirt-node in terms of oVirt?) locally; 4. To write hook script that will provide the comparation of existing checksums and initial checksums every time when VM is starting. //vdsm hooks, libguestfs So you can see that libguestfs is a base tool for this part of system but in some cases it cannot work with disk images properly (I described them here: https://bugzilla.redhat.com/show_bug.cgi?id=1053684, https://www.redhat.com/archives/libguestfs/2014-January/msg00175.html). Probably I use the library in the wrong way and this causes all the issues… Honestly speaking I don’t know if the functionality that I have just described should integrate with oVirt or it rather should be a standalone solution. I would be glad if you could comment this problem. Thank you. Sincerely, Vitaly Isaev Software engineer Information security department Fintech JSC, Moscow, Russia