virt-df stuck at supermin: [...] waiting for /sys/block/sdb/dev to appear
This problem occurs on certain machines but I can't tell why and how exactly to fix it. The message suggests a problem with kernel modules, but they all seem to load successfully, including many virtio modules. Best Regards, Peter
On Tue, Nov 20, 2018 at 03:15:54PM +0200, Peter Dimitrov wrote:
This problem occurs on certain machines but I can't tell why and how exactly to fix it.
The message suggests a problem with kernel modules, but they all seem to load successfully, including many virtio modules.
These are quite old versions of kernel/qemu/libguestfs/supermin so I can only really offer general advice. You are right that the problem is caused because the appliance kernel does not see any virtio-scsi disks. That can be for a variety of reasons such as: - The virtio drivers in the appliance are not loaded or not working for some reason. - qemu isn't presenting virtio devices to the guest.
From the trace:
libguestfs: libvirt XML:\n [...] <devices> <controller type="scsi" index="0" model="virtio-scsi"/> <disk device="disk" type="file"> <source file="/tmp/libguestfsdTUUkY/overlay1"> <seclabel model="selinux" relabel="no"/> </source> <target dev="sda" bus="scsi"/> <driver name="qemu" type="qcow2" cache="unsafe"/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> </disk> <disk type="file" device="disk"> <source file="/tmp/libguestfsdTUUkY/overlay2"/> <target dev="sdb" bus="scsi"/> <driver name="qemu" type="qcow2" cache="unsafe"/> <address type="drive" controller="0" bus="0" target="1" unit="0"/> <shareable/> </disk> This ought to cause libvirt to tell qemu to present virtio-scsi devices. You can check if they are actually present on the qemu command line by looking for /var/log/libvirt/qemu/guestfs-XXX.log supermin: internal insmod crc32c-intel.ko supermin: internal insmod virtio.ko supermin: internal insmod virtio_ring.ko supermin: internal insmod virtio_blk.ko supermin: internal insmod virtio_console.ko supermin: internal insmod virtio_net.ko supermin: internal insmod virtio_scsi.ko The Virtio SCSI driver is getting loaded. supermin: internal insmod virtio_balloon.ko supermin: internal insmod virtio_input.ko supermin: internal insmod virtio_mmio.ko supermin: internal insmod virtio_pci.ko [ 0.289514] virtio-pci 0000:00:02.0: PCI->APIC IRQ transform: INT A -> IRQ 10 [ 0.289888] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy driver [ 0.290355] virtio-pci 0000:00:03.0: PCI->APIC IRQ transform: INT A -> IRQ 11 [ 0.290768] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver [ 0.291008] virtio-pci 0000:00:04.0: PCI->APIC IRQ transform: INT A -> IRQ 11 [ 0.291392] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver This indicates that the Virtio PCI transport module is being loaded. The "virtio_pci: leaving for legacy driver" message is worrying. I think what's happening here is your appliance kernel only has drivers for virtio 1.0 ("virtio modern") but your qemu is presenting virtio legacy drivers. There's a kernel config option to enable virtio legacy (VIRTIO_PCI_LEGACY) which you should probably try first. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Seems like the devices are present: LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none TMPDIR=/var/tmp /usr/bin/qemu-kvm -name guestfs-f9uckq9gdkacg5hn,debug-threads=on -S -machine pc-i440fx-2.6,accel=kvm,usb=off -cpu host -m 500 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 4e89287f-6f82-4f54-9808-43527e9d3c91 -nographic -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3-guestfs-f9uckq9gdkac/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-reboot -no-acpi -boot strict=on -kernel /var/tmp/.guestfs-0/appliance.d/kernel -initrd /var/tmp/.guestfs-0/appliance.d/initrd -append 'panic=1 console=ttyS0 udevtimeout=6000 udev.event-timeout=6000 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color' -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file= */tmp/libguestfspS25nG/overlay1*,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive file=*/tmp/libguestfspS25nG/overlay2*,format=qcow2,if=none,id=drive-scsi0-0-1-0,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0 -chardev socket,id=charserial0,path=/tmp/libguestfspS25nG/console.sock -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/tmp/libguestfspS25nG/guestfsd.sock -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on Domain id=3 is tainted: high-privileges Domain id=3 is tainted: custom-argv Domain id=3 is tainted: host-cpu 2018-11-20T15:17:14.956988Z qemu-system-x86_64: terminating on signal 15 from pid 6001 2018-11-20 15:17:15.158+0000: shutting down Compiling kernel with VIRTIO_PCI_LEGACY=m fixes it the issue! Best Regards, Peter On Tue, Nov 20, 2018 at 5:24 PM Richard W.M. Jones <rjones@redhat.com> wrote:
On Tue, Nov 20, 2018 at 03:15:54PM +0200, Peter Dimitrov wrote:
This problem occurs on certain machines but I can't tell why and how exactly to fix it.
The message suggests a problem with kernel modules, but they all seem to load successfully, including many virtio modules.
These are quite old versions of kernel/qemu/libguestfs/supermin so I can only really offer general advice. You are right that the problem is caused because the appliance kernel does not see any virtio-scsi disks. That can be for a variety of reasons such as:
- The virtio drivers in the appliance are not loaded or not working for some reason.
- qemu isn't presenting virtio devices to the guest.
From the trace:
libguestfs: libvirt XML:\n [...] <devices> <controller type="scsi" index="0" model="virtio-scsi"/> <disk device="disk" type="file"> <source file="/tmp/libguestfsdTUUkY/overlay1"> <seclabel model="selinux" relabel="no"/> </source> <target dev="sda" bus="scsi"/> <driver name="qemu" type="qcow2" cache="unsafe"/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> </disk> <disk type="file" device="disk"> <source file="/tmp/libguestfsdTUUkY/overlay2"/> <target dev="sdb" bus="scsi"/> <driver name="qemu" type="qcow2" cache="unsafe"/> <address type="drive" controller="0" bus="0" target="1" unit="0"/> <shareable/> </disk>
This ought to cause libvirt to tell qemu to present virtio-scsi devices. You can check if they are actually present on the qemu command line by looking for /var/log/libvirt/qemu/guestfs-XXX.log
supermin: internal insmod crc32c-intel.ko supermin: internal insmod virtio.ko supermin: internal insmod virtio_ring.ko supermin: internal insmod virtio_blk.ko supermin: internal insmod virtio_console.ko supermin: internal insmod virtio_net.ko supermin: internal insmod virtio_scsi.ko
The Virtio SCSI driver is getting loaded.
supermin: internal insmod virtio_balloon.ko supermin: internal insmod virtio_input.ko supermin: internal insmod virtio_mmio.ko supermin: internal insmod virtio_pci.ko [ 0.289514] virtio-pci 0000:00:02.0: PCI->APIC IRQ transform: INT A -> IRQ 10 [ 0.289888] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy driver [ 0.290355] virtio-pci 0000:00:03.0: PCI->APIC IRQ transform: INT A -> IRQ 11 [ 0.290768] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver [ 0.291008] virtio-pci 0000:00:04.0: PCI->APIC IRQ transform: INT A -> IRQ 11 [ 0.291392] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
This indicates that the Virtio PCI transport module is being loaded. The "virtio_pci: leaving for legacy driver" message is worrying.
I think what's happening here is your appliance kernel only has drivers for virtio 1.0 ("virtio modern") but your qemu is presenting virtio legacy drivers. There's a kernel config option to enable virtio legacy (VIRTIO_PCI_LEGACY) which you should probably try first.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
participants (2)
-
Peter Dimitrov -
Richard W.M. Jones