Hi, I've been trying to migrate a physical host with two disks (the first, sda, is the OS disk and the second, sdb, is a data disk) into a VM, and it always fails on the second disk. The first disk, the OS disk, migrates fine, but when it starts on the second it fails almost straightaway with the following error: (0.00/100%)^Mqemu-img: Could not open '/var/tmp/v2vovl811e67.qcow2': Could not open backing file: Failed to connect socket: Connection refused I can't see this backing file, but it may be deleted when the process fails, as unfortunately it cleans up and deletes not only the successfully migrated sda and associated files, but I suspect the second as well. Most of the rest of the log can be seen at: http://theninthdimension.blogspot.co.uk/2017/02/virt-p2v-error.html The conversion host has plenty of disk space in the migration path: the image of sdb is about 700GB, and it has 5TB free. I can always successfully migrate just sda. I can't just choose sdb as it needs an OS disk included. Details of the hosts being migrated: RHEL 5 x86_64 Details of the conversion server: Fedora 23 with virt-v2v-1.32.10-1.fc23.x86_64, libvirt-1.2.18.4-1.fc23.x86_64 The boot image was created on the conversion host using both Fedora 23 and CentOS 7.3 (both see the same issue) Any ideas? Thanks, Cam
On Wed, Feb 01, 2017 at 04:57:24PM +0000, cmc wrote:
Hi,
I've been trying to migrate a physical host with two disks (the first, sda, is the OS disk and the second, sdb, is a data disk) into a VM, and it always fails on the second disk. The first disk, the OS disk, migrates fine, but when it starts on the second it fails almost straightaway with the following error:
(0.00/100%)^Mqemu-img: Could not open '/var/tmp/v2vovl811e67.qcow2': Could not open backing file: Failed to connect socket: Connection refused
I can't see this backing file, but it may be deleted when the process fails, as unfortunately it cleans up and deletes not only the successfully migrated sda and associated files, but I suspect the second as well. Most of the rest of the log can be seen at:
http://theninthdimension.blogspot.co.uk/2017/02/virt-p2v-error.html
What's actually happening is when the conversion server tries to connect to the second disk, the ssh connection has been dropped, hence the "backing file" (in fact an NBD server) cannot be connected to. Refer to the second diagram here: http://libguestfs.org/virt-p2v.1.html#how-virt-p2v-works I don't know why the second connection has been dropped. Does the conversion server have ssh timeouts enabled? Or bash timeouts? You can check ClientAlive* settings in /etc/ssh/sshd_config. Also look for TMOUT or TIMEOUT environment variables on the conversion server. You can also look in the logs on the conversion server to see if the ssh connection was dropped or forced to close, and why. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Ok, thanks Rich, I will take a look and get back to you. On Wed, Feb 1, 2017 at 5:13 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Feb 01, 2017 at 04:57:24PM +0000, cmc wrote:
Hi,
I've been trying to migrate a physical host with two disks (the first, sda, is the OS disk and the second, sdb, is a data disk) into a VM, and it always fails on the second disk. The first disk, the OS disk, migrates fine, but when it starts on the second it fails almost straightaway with the following error:
(0.00/100%)^Mqemu-img: Could not open '/var/tmp/v2vovl811e67.qcow2': Could not open backing file: Failed to connect socket: Connection refused
I can't see this backing file, but it may be deleted when the process fails, as unfortunately it cleans up and deletes not only the successfully migrated sda and associated files, but I suspect the second as well. Most of the rest of the log can be seen at:
http://theninthdimension.blogspot.co.uk/2017/02/virt-p2v-error.html
What's actually happening is when the conversion server tries to connect to the second disk, the ssh connection has been dropped, hence the "backing file" (in fact an NBD server) cannot be connected to.
Refer to the second diagram here:
http://libguestfs.org/virt-p2v.1.html#how-virt-p2v-works
I don't know why the second connection has been dropped. Does the conversion server have ssh timeouts enabled? Or bash timeouts? You can check ClientAlive* settings in /etc/ssh/sshd_config. Also look for TMOUT or TIMEOUT environment variables on the conversion server. You can also look in the logs on the conversion server to see if the ssh connection was dropped or forced to close, and why.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Looking on our firewall, it will timeout inactive connections after 15 minutes of inactivity. I'm guessing there is no keepalive in the control connections. Thanks, Cam On Wed, Feb 1, 2017 at 6:38 PM, cmc <iucounu@gmail.com> wrote:
Ok, thanks Rich, I will take a look and get back to you.
On Wed, Feb 1, 2017 at 5:13 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Feb 01, 2017 at 04:57:24PM +0000, cmc wrote:
Hi,
I've been trying to migrate a physical host with two disks (the first, sda, is the OS disk and the second, sdb, is a data disk) into a VM, and it always fails on the second disk. The first disk, the OS disk, migrates fine, but when it starts on the second it fails almost straightaway with the following error:
(0.00/100%)^Mqemu-img: Could not open '/var/tmp/v2vovl811e67.qcow2': Could not open backing file: Failed to connect socket: Connection refused
I can't see this backing file, but it may be deleted when the process fails, as unfortunately it cleans up and deletes not only the successfully migrated sda and associated files, but I suspect the second as well. Most of the rest of the log can be seen at:
http://theninthdimension.blogspot.co.uk/2017/02/virt-p2v-error.html
What's actually happening is when the conversion server tries to connect to the second disk, the ssh connection has been dropped, hence the "backing file" (in fact an NBD server) cannot be connected to.
Refer to the second diagram here:
http://libguestfs.org/virt-p2v.1.html#how-virt-p2v-works
I don't know why the second connection has been dropped. Does the conversion server have ssh timeouts enabled? Or bash timeouts? You can check ClientAlive* settings in /etc/ssh/sshd_config. Also look for TMOUT or TIMEOUT environment variables on the conversion server. You can also look in the logs on the conversion server to see if the ssh connection was dropped or forced to close, and why.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
with regard to ssh timeouts on the conversion server, there weren't any set, nor were there any timeout env variables. I could not see any ssh drops in /var/log/messages, or anything in the vdsm log Thanks On Fri, Feb 3, 2017 at 7:29 PM, cmc <iucounu@gmail.com> wrote:
Looking on our firewall, it will timeout inactive connections after 15 minutes of inactivity. I'm guessing there is no keepalive in the control connections.
Thanks,
Cam
On Wed, Feb 1, 2017 at 6:38 PM, cmc <iucounu@gmail.com> wrote:
Ok, thanks Rich, I will take a look and get back to you.
On Wed, Feb 1, 2017 at 5:13 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
On Wed, Feb 01, 2017 at 04:57:24PM +0000, cmc wrote:
Hi,
I've been trying to migrate a physical host with two disks (the first, sda, is the OS disk and the second, sdb, is a data disk) into a VM, and it always fails on the second disk. The first disk, the OS disk, migrates fine, but when it starts on the second it fails almost straightaway with the following error:
(0.00/100%)^Mqemu-img: Could not open '/var/tmp/v2vovl811e67.qcow2': Could not open backing file: Failed to connect socket: Connection refused
I can't see this backing file, but it may be deleted when the process fails, as unfortunately it cleans up and deletes not only the successfully migrated sda and associated files, but I suspect the second as well. Most of the rest of the log can be seen at:
http://theninthdimension.blogspot.co.uk/2017/02/virt-p2v-error.html
What's actually happening is when the conversion server tries to connect to the second disk, the ssh connection has been dropped, hence the "backing file" (in fact an NBD server) cannot be connected to.
Refer to the second diagram here:
http://libguestfs.org/virt-p2v.1.html#how-virt-p2v-works
I don't know why the second connection has been dropped. Does the conversion server have ssh timeouts enabled? Or bash timeouts? You can check ClientAlive* settings in /etc/ssh/sshd_config. Also look for TMOUT or TIMEOUT environment variables on the conversion server. You can also look in the logs on the conversion server to see if the ssh connection was dropped or forced to close, and why.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Fri, Feb 03, 2017 at 07:29:07PM +0000, cmc wrote:
Looking on our firewall, it will timeout inactive connections after 15 minutes of inactivity.
Yes, this will likely break virt-p2v.
I'm guessing there is no keepalive in the control connections.
The default for ssh is to use TCP keepalives. However those are very infrequent (once every 2 hours IIRC), so it won't prevent a firewall with such a short timeout from breaking virt-p2v. Easiest thing here would be to fix the firewall. 15 minutes is a very short timeout. Doesn't that cause all kinds of problems for regular SSH usage? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
On Sat, 4 Feb 2017 10:15:57 +0000 "Richard W.M. Jones" <rjones@redhat.com> wrote:
On Fri, Feb 03, 2017 at 07:29:07PM +0000, cmc wrote:
Looking on our firewall, it will timeout inactive connections after 15 minutes of inactivity.
Yes, this will likely break virt-p2v.
I'm guessing there is no keepalive in the control connections.
The default for ssh is to use TCP keepalives. However those are very infrequent (once every 2 hours IIRC), so it won't prevent a firewall with such a short timeout from breaking virt-p2v.
Easiest thing here would be to fix the firewall. 15 minutes is a very short timeout. Doesn't that cause all kinds of problems for regular SSH usage?
SSH client has option ServerAliveInterval (in seconds) that instructs SSH to send, after period of inactivity, a message through control connection and request a response from server. The default is 0, i.e. no such messages are sent. The intent is mostly to detect early when connection was dropped, but it should also help you with your firewall. Hope this helps, Tomas
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
_______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
-- Tomáš Golembiovský <tgolembi@redhat.com>
On Sat, Feb 04, 2017 at 02:10:30PM +0100, Tomáš Golembiovský wrote:
On Sat, 4 Feb 2017 10:15:57 +0000 "Richard W.M. Jones" <rjones@redhat.com> wrote:
On Fri, Feb 03, 2017 at 07:29:07PM +0000, cmc wrote:
Looking on our firewall, it will timeout inactive connections after 15 minutes of inactivity.
Yes, this will likely break virt-p2v.
I'm guessing there is no keepalive in the control connections.
The default for ssh is to use TCP keepalives. However those are very infrequent (once every 2 hours IIRC), so it won't prevent a firewall with such a short timeout from breaking virt-p2v.
Easiest thing here would be to fix the firewall. 15 minutes is a very short timeout. Doesn't that cause all kinds of problems for regular SSH usage?
SSH client has option ServerAliveInterval (in seconds) that instructs SSH to send, after period of inactivity, a message through control connection and request a response from server. The default is 0, i.e. no such messages are sent. The intent is mostly to detect early when connection was dropped, but it should also help you with your firewall.
Hope this helps,
Indeed that does help with most things (except for shell-level timeouts). I'll post a patch in a bit. Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
participants (3)
-
cmc -
Richard W.M. Jones -
Tomáš Golembiovský