10:41 a.m.
---------- Forwarded message ----------
From: Skippy VonDrake <skippyvondrake(a)gmail.com>
Date: Tue, Apr 9, 2013 at 11:38 AM
Subject: Re: [Qemu-devel] debugging qemu-nbd.c
To: "Richard W.M. Jones" <rjones(a)redhat.com>
> >> I have an app that forks a process with calls qemu-nbd
with an offset
> >> into an image.
> >> The app then uses libguestfs to attach the nbd device.
> >> But it appears that qemu is failing.
> >
> > When you say "qemu is failing", do you mean qemu-nbd, or the qemu
> > subprocess that libguestfs uses? Does it print any error?
>
> I believe it is the qemu subprocess that libguestfs is using.
> I'm basing that assumption on the log file statements from both libguestfs
> and qemu, and from stderr from libguestfs.
>
> The only stderr from libguestfs is "child_cleanup:... child process died"
Enable debugging in libguestfs to find out what's going on:
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
With both LIBGUESTFS_DEBUG and LIBGUESTFS_TRACE set and
guestfs_set_verbose(g,1) and guestfs_set_trace(g,1).
Below is the output.
Some messages may look unfamiliar because I placed some
statements in libguestfs just for a 'crude' tracing into the library.
The initial 'nbd_client_thread' message is one I placed in the qemu sources.
executing 'system(sudo rmmod nbd)
executing 'system(sudo modprobe nbd max_part=16)
setupNbdServer - size: 77824
setupNbdServer - system(serverCmd) string: qemu-nbd -v -n -o 77824 -t
-p 3000 -c /dev/nbd0 /home/skippy/mnt/mnt0/win7x64amdHP.vdi
nbd_client_thread:line 260::12806|939447:: qemu-NBD device /dev/nbd0
is NOWWW connected to /home/skippy/mnt/mnt0/win7x64amdHP.vdi
libguestfs: trace: add_drive "" "format:raw" "protocol:nbd"
"server:localhost:3000"
libguestfs: trace: add_drive = 0
guestfs is launching the appliance
libguestfs: trace: launch
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: launch: attach-method=appliance
libguestfs: launch: tmpdir=/tmp/libguestfsscRysl
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: [00000ms] begin testing qemu features
libguestfs: command: run:
/home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -help
libguestfs: command: run:
/home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -version
libguestfs: qemu version 1.4
libguestfs: command: run:
/home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -machine accel=kvm:tcg
libguestfs: command: run: \ -device ?
libguestfs: [00025ms] finished testing qemu features
libguestfs: [00025ms] r from fork: 13658 \n
\ libguestfs: error: appliance closed the connection unexpectedly,
see earlier error messages
libguestfs: child_cleanup: 0x11f4ba0: child process died
libguestfs: sending SIGTERM to process 13658
libguestfs: error:
/home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
exited with error status 1, see debug messages above
libguestfs: error: guestfs_launch failed, see earlier error messages
libguestfs: trace: launch = -1 (error)
ERROR from setupGuestfs-> guestfs_launch, errnum: 0
Last error msg: guestfs_launch failed, see earlier error messages
main thread exiting
libguestfs: trace: close
libguestfs: closing guestfs handle 0x11f4ba0 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsscRysl
As you can see, the output looks similar to my previous post. At which you
suggested that qemu might be failing. So that is when I started trying to
pinpoint the problem in qemu. And the 'chdir' command was the final point
at which my file-log statements ended. For that logging I'd open the log,
write a string and close the log each time.
One bit that might be of interest - my app does a fork/system command
to launch the ndb server. When I give the disconnect command (after the
error output above), I see this displayed from qemu:
$ nbd.c:nbd_trip():L1106: From: 18446744073709551104, Len: 0, Size:
20003880960, Offset: 77824
nbd.c:nbd_trip():L1107: requested operation past EOF--bad client?
The image is 20GB with a VDI header prepended to it. The size of
the header is displayed with "setupNbdServer - size: 77824" and
used for the offset parameter for qemu-nbd.
I don't know what to make of the "From" value nor the 'Len' of zero
in the last output.
11:36 a.m.
On Tue, Apr 09, 2013 at 11:41:15AM -0400, Skippy VonDrake wrote:
libguestfs: qemu version 1.4
libguestfs: command: run:
/home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -machine accel=kvm:tcg
libguestfs: command: run: \ -device ?
libguestfs: [00025ms] finished testing qemu features
libguestfs: [00025ms] r from fork: 13658 \n
\ libguestfs: error: appliance closed the connection unexpectedly,
see earlier error messages
Either some output is missing, or something very strange is happening
in the appliance back end. Would advise adding lots more debugging to
the code that starts at this comment:
if (r == 0) { /* Child (qemu). */
I don't think the error has anything to do with qemu-nbd. It doesn't
even get to the point where libguestfs prints the qemu command line.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
1:25 p.m.
> libguestfs: qemu version 1.4
> libguestfs: command: run:
> /home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
> libguestfs: command: run: \ -nographic
> libguestfs: command: run: \ -machine accel=kvm:tcg
> libguestfs: command: run: \ -device ?
> libguestfs: [00025ms] finished testing qemu features
> libguestfs: [00025ms] r from fork: 13658 \n
> \ libguestfs: error: appliance closed the connection unexpectedly,
> see earlier error messages
Either some output is missing, or something very strange is happening
in the appliance back end. Would advise adding lots more debugging to
the code that starts at this comment:
if (r == 0) { /* Child (qemu). */
I don't think the error has anything to do with qemu-nbd. It doesn't
even get to the point where libguestfs prints the qemu command line.
Maybe I mucked up libguestfs logging with my own logging... don't see
how but it's possible.
Here are the log statements I put in for the qemu-child.
I ended with listing the args in g->app.cmdline just before
this statement:
execv (g->qemu, g->app.cmdline); /* Run qemu. */
in guestfs forked child - pid for qemu process: 29439
qemu child - before if (has_appliance_drive)
qemu child - before Set up virtio-serial
qemu child - before Dump the command line (after setting up stderr)
qemu child - before * Put qemu in a new process group. *
/usr/local/bin/qemu-system-x86_64
-global
virtio-blk-pci.scsi=off
-nodefconfig
-nodefaults
-nographic
-device
virtio-scsi-pci,id=scsi
-drive
file=nbd:localhost:3000,format=raw,id=hd0,if=none
-device
scsi-hd,drive=hd0
-drive
file=../appliance/root,snapshot=on,id=appliance,if=none,cache=unsafe
-device
scsi-hd,drive=appliance
-machine
accel=kvm:tcg
-m
500
-no-reboot
-no-hpet
-device
virtio-serial
-serial
stdio
-device
sga
-chardev
socket,path=/tmp/libguestfsdeuFRl/guestfsd.sock,id=channel0
-device
virtserialport,chardev=channel0,name=org.libguestfs.channel.0
-kernel
../appliance/kernel
-initrd
../appliance/initrd
-append
panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off
printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0
guestfs_verbose=1 TERM=xterm
-Skippy
4:11 a.m.
On Tue, Apr 09, 2013 at 02:25:28PM -0400, Skippy VonDrake wrote:
>> libguestfs: qemu version 1.4
>> libguestfs: command: run:
>> /home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
>> libguestfs: command: run: \ -nographic
>> libguestfs: command: run: \ -machine accel=kvm:tcg
>> libguestfs: command: run: \ -device ?
>> libguestfs: [00025ms] finished testing qemu features
>> libguestfs: [00025ms] r from fork: 13658 \n
>> \ libguestfs: error: appliance closed the connection unexpectedly,
>> see earlier error messages
>
> Either some output is missing, or something very strange is happening
> in the appliance back end. Would advise adding lots more debugging to
> the code that starts at this comment:
>
> if (r == 0) { /* Child (qemu). */
>
> I don't think the error has anything to do with qemu-nbd. It doesn't
> even get to the point where libguestfs prints the qemu command line.
Maybe I mucked up libguestfs logging with my own logging... don't see
how but it's possible.
Here are the log statements I put in for the qemu-child.
I ended with listing the args in g->app.cmdline just before
this statement:
execv (g->qemu, g->app.cmdline); /* Run qemu. */
in guestfs forked child - pid for qemu process: 29439
qemu child - before if (has_appliance_drive)
qemu child - before Set up virtio-serial
qemu child - before Dump the command line (after setting up stderr)
qemu child - before * Put qemu in a new process group. *
/usr/local/bin/qemu-system-x86_64
-global
virtio-blk-pci.scsi=off
-nodefconfig
-nodefaults
-nographic
-device
virtio-scsi-pci,id=scsi
-drive
file=nbd:localhost:3000,format=raw,id=hd0,if=none
My next guess was going to be that it was failing to create
the nbd: command line parameter properly, but this looks OK.
I would enable core dumps (ulimit -c unlimited) and then see what
process dumps core and get a stack trace from that.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
4274
days inactive
4275
days old
3 comments
2 participants
participants (2)
-
Richard W.M. Jones -
Skippy VonDrake