On Wed, Jan 24, 2018 at 11:49:13PM +0100, Luca 'remix_tj' Lorenzetto wrote: > Hello, > > i've started my migrations from vmware today. I had successfully > migrated over 200 VM from vmware to another cluster based on 4.0 using > our home-made scripts interacting with the API's. All the migrated vms > are running RHEL 6 or 7, with no SELinux. > > We understood a lot about the necessities and we recorded also some > metrics about migration times. In July, with 4.0 as destination, we > were migrating ~30gb vm in ~40 mins. > It was an acceptable time, considering that about 50% of our vms stand > around that size. > > Today we started migrating to the production cluster, that is, > instead, running 4.1.8. With the same scripts, the same api calls, and > a vm of about 50gb we were supposing that we will have the vm running > in the new cluster after 70 minutes, more or less. > > Instead, the migration is taking more than 2 hours, and this not > because of the slow conversion time by qemu-img given that we're > transferring an entire disk via http. > Looking at the log, seems that activities executed before qemu-img > took more than 2000 seconds. As example, appears to me that dracut > took more than 14 minutes, which is in my opinion a bit long. There's got to be some difference between your staging environment and your production environment, and I'm pretty sure it has nothing to do with the version of oVirt. Are you running virt-v2v inside a virtual machine, and previously you ran it on bare-metal? Or did you disable nested KVM? That seems like the most likely explanation for the difference (although I'm surprised that the difference is so large).

> Is there any option to get a quicker conversion? Also some tasks to > run in the guests before the conversion are accepted. > > We have to migrate ~300 vms in 2.5 months, and we're only at 11 after > 7 hours (and today an exception that allowed us to start 4 hours in > advance, but usually our maintenance time is significantly lower). > > This is a filtered out log reporting only the rows were we can > understand how much time has passed: > > [ 0.0] Opening the source -i libvirt -ic > vpx://vmwareuser%40domain@vcenter/DC/Cluster/Host?no_verify=1 > vmtoconvert > [ 6.1] Creating an overlay to protect the source from being modified > [ 7.4] Initializing the target -o vdsm -os > /rhev/data-center/e8263fb4-114d-4706-b1c0-5defcd15d16b/a118578a-4cf2-4e0c-ac47-20e9f0321da1 > --vdsm-image-uuid 1a93e503-ce57-4631-8dd2-eeeae45866ca --vdsm-vol-uuid > 88d92582-0f53-43b0-89ff-af1c17ea8618 --vdsm-vm-uuid > 1434e14f-e228-41c1-b769-dcf48b258b12 --vdsm-ovf-output > /var/run/vdsm/v2v > [ 7.4] Opening the overlay > [00034ms] /usr/libexec/qemu-kvm \ > [ 0.000000] Initializing cgroup subsys cpu > [ 0.000000] Initializing cgroup subsys cpuacct > [ 0.000000] Linux version 3.10.0-693.11.1.el7.x86_64 > (mockbuild(a)x86-041.build.eng.bos.redhat.com) (gcc version 4.8.5 > 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Fri Oct 27 05:39:05 EDT > 2017 > [ 0.000000] Command line: panic=1 console=ttyS0 edd=off > udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 > cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable > 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 > guestfs_network=1 TERM=linux guestfs_identifier=v2v > [ 0.000000] e820: BIOS-provided physical RAM map: > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f7ff] usable > [ 0.000000] BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved > [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007cfddfff] usable > [ 0.000000] BIOS-e820: [mem 0x000000007cfde000-0x000000007cffffff] reserved > [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved > [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved > [ 0.000000] NX (Execute Disable) protection: active > [ 0.000000] SMBIOS 2.8 present. > [ 0.000000] Hypervisor detected: KVM > [ 0.000000] e820: last_pfn = 0x7cfde max_arch_pfn = 0x400000000 > [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 > [ 0.000000] found SMP MP-table at [mem 0x000f72f0-0x000f72ff] > mapped at [ffff8800000f72f0] > [ 0.000000] Using GB pages for direct mapping > [ 0.000000] RAMDISK: [mem 0x7ccb2000-0x7cfcffff] > [ 0.000000] Early table checksum verification disabled > [ 0.000000] ACPI: RSDP 00000000000f70d0 00014 (v00 BOCHS ) > [ 0.000000] ACPI: RSDT 000000007cfe14d5 0002C (v01 BOCHS BXPCRSDT > 00000001 BXPC 00000001) > [ 0.000000] ACPI: FACP 000000007cfe13e9 00074 (v01 BOCHS BXPCFACP > 00000001 BXPC 00000001) > [ 0.000000] ACPI: DSDT 000000007cfe0040 013A9 (v01 BOCHS BXPCDSDT > 00000001 BXPC 00000001) > [ 0.000000] ACPI: FACS 000000007cfe0000 00040 > [ 0.000000] ACPI: APIC 000000007cfe145d 00078 (v01 BOCHS BXPCAPIC > 00000001 BXPC 00000001) > [ 0.000000] No NUMA configuration found > [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007cfddfff] > [ 0.000000] NODE_DATA(0) allocated [mem 0x7cc8b000-0x7ccb1fff] > [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 > [ 0.000000] kvm-clock: cpu 0, msr 0:7cc3b001, primary cpu clock > [ 0.000000] kvm-clock: using sched offset of 1030608733 cycles > [ 0.000000] Zone ranges: > [ 0.000000] DMA [mem 0x00001000-0x00ffffff] > [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] > [ 0.000000] Normal empty > [ 0.000000] Movable zone start for each node > [ 0.000000] Early memory node ranges > [ 0.000000] node 0: [mem 0x00001000-0x0009efff] > [ 0.000000] node 0: [mem 0x00100000-0x7cfddfff] > [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x7cfddfff] > [ 0.000000] ACPI: PM-Timer IO Port: 0x608 > [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) > [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) > [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) > [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) > [ 0.000000] Using ACPI (MADT) for SMP configuration information > [ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs > [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] > [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] > [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] > [ 0.000000] e820: [mem 0x7d000000-0xfeffbfff] available for PCI devices > [ 0.000000] Booting paravirtualized kernel on KVM > [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 > nr_cpu_ids:1 nr_node_ids:1 > [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007ca00000 s97048 > r8192 d29928 u2097152 > [ 0.000000] KVM setup async PF for cpu 0 > [ 0.000000] kvm-stealtime: cpu 0, msr 7ca0f440 > [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) > [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. > Total pages: 503847 > [ 0.000000] Policy zone: DMA32 > [ 0.000000] Kernel command line: panic=1 console=ttyS0 edd=off > udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 > cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable > 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 > guestfs_network=1 TERM=linux guestfs_identifier=v2v > [ 0.000000] Disabling memory control group subsystem > [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) > [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 > [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using > standard form > [ 0.000000] Memory: 1994224k/2047864k available (6886k kernel code, > 392k absent, 53248k reserved, 4545k data, 1764k init) > [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 > [ 0.000000] Hierarchical RCU implementation. > [ 0.000000] \tRCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1. > [ 0.000000] NR_IRQS:327936 nr_irqs:256 0 > [ 0.000000] Console: colour *CGA 80x25 > [ 0.000000] console [ttyS0] enabled > [ 0.000000] tsc: Detected 2099.998 MHz processor > [ 0.065500] Calibrating delay loop (skipped) preset value.. 4199.99 > BogoMIPS (lpj=2099998) > [ 0.066153] pid_max: default: 32768 minimum: 301 > [ 0.066548] Security Framework initialized > [ 0.066872] SELinux: Disabled at boot. > [ 0.067181] Yama: becoming mindful. > [ 0.067622] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) > [ 0.068574] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) > [ 0.069290] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes) > [ 0.069813] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes) > [ 0.070525] Initializing cgroup subsys memory > [ 0.070877] Initializing cgroup subsys devices > [ 0.071237] Initializing cgroup subsys freezer > [ 0.071589] Initializing cgroup subsys net_cls > [ 0.071932] Initializing cgroup subsys blkio > [ 0.072275] Initializing cgroup subsys perf_event > [ 0.072644] Initializing cgroup subsys hugetlb > [ 0.072984] Initializing cgroup subsys pids > [ 0.073316] Initializing cgroup subsys net_prio > [ 0.073721] CPU: Physical Processor ID: 0 > [ 0.074810] mce: CPU supports 10 MCE banks > [ 0.075185] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 > [ 0.075621] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 > [ 0.076030] tlb_flushall_shift: 6 > [ 0.085827] Freeing SMP alternatives: 24k freed > [ 0.091125] ACPI: Core revision 20130517 > [ 0.091976] ACPI: All ACPI Tables successfully acquired > [ 0.092448] ftrace: allocating 26586 entries in 104 pages > [ 0.116144] smpboot: Max logical packages: 1 > [ 0.116640] Enabling x2apic > [ 0.116863] Enabled x2apic > [ 0.117290] Switched APIC routing to physical x2apic. > [ 0.118588] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 > [ 0.119054] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2683 v4 @ > 2.10GHz (fam: 06, model: 4f, stepping: 01) > [ 0.119813] Performance Events: 16-deep LBR, Broadwell events, > Intel PMU driver. > [ 0.121545] ... version: 2 > [ 0.121847] ... bit width: 48 > [ 0.122161] ... generic registers: 4 > [ 0.122472] ... value mask: 0000ffffffffffff > [ 0.122874] ... max period: 000000007fffffff > [ 0.123276] ... fixed-purpose events: 3 > [ 0.123584] ... event mask: 000000070000000f > [ 0.124004] KVM setup paravirtual spinlock > [ 0.125379] Brought up 1 CPUs > [ 0.125616] smpboot: Total of 1 processors activated (4199.99 BogoMIPS) > [ 0.126464] devtmpfs: initialized > [ 0.128347] EVM: security.selinux > [ 0.128608] EVM: security.ima > [ 0.128835] EVM: security.capability > [ 0.129796] atomic64 test passed for x86-64 platform with CX8 and with SSE > [ 0.130333] pinctrl core: initialized pinctrl subsystem > [ 0.130805] RTC time: 20:26:38, date: 01/24/18 > [ 0.131217] NET: Registered protocol family 16 > [ 0.131774] ACPI: bus type PCI registered > [ 0.132096] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 > [ 0.132660] PCI: Using configuration type 1 for base access > [ 0.133830] ACPI: Added _OSI(Module Device) > [ 0.134170] ACPI: Added _OSI(Processor Device) > [ 0.134514] ACPI: Added _OSI(3.0 _SCP Extensions) > [ 0.134872] ACPI: Added _OSI(Processor Aggregator Device) > [ 0.137001] ACPI: Interpreter enabled > [ 0.137303] ACPI: (supports S0 S5) > [ 0.137573] ACPI: Using IOAPIC for interrupt routing > [ 0.137971] PCI: Using host bridge windows from ACPI; if necessary, > use "pci=nocrs" and report a bug > [ 0.140442] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) > [ 0.140917] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] > [ 0.141446] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM > [ 0.141961] acpi PNP0A03:00: fail to add MMCONFIG information, > can't access extended PCI configuration space under this bridge. > [ 0.142997] acpiphp: Slot [2] registered > [ 0.143309] acpiphp: Slot [3] registered > [ 0.143625] acpiphp: Slot [4] registered > [ 0.143949] acpiphp: Slot [5] registered > [ 0.144260] acpiphp: Slot [6] registered > [ 0.144575] acpiphp: Slot [7] registered > [ 0.144887] acpiphp: Slot [8] registered > [ 0.145205] acpiphp: Slot [9] registered > [ 0.145523] acpiphp: Slot [10] registered > [ 0.145841] acpiphp: Slot [11] registered > [ 0.146161] acpiphp: Slot [12] registered > [ 0.146642] acpiphp: Slot [13] registered > [ 0.146960] acpiphp: Slot [14] registered > [ 0.147279] acpiphp: Slot [15] registered > [ 0.147602] acpiphp: Slot [16] registered > [ 0.147934] acpiphp: Slot [17] registered > [ 0.148255] acpiphp: Slot [18] registered > [ 0.148579] acpiphp: Slot [19] registered > [ 0.148896] acpiphp: Slot [20] registered > [ 0.149219] acpiphp: Slot [21] registered > [ 0.149546] acpiphp: Slot [22] registered > [ 0.149863] acpiphp: Slot [23] registered > [ 0.150178] acpiphp: Slot [24] registered > [ 0.150505] acpiphp: Slot [25] registered > [ 0.150824] acpiphp: Slot [26] registered > [ 0.151139] acpiphp: Slot [27] registered > [ 0.151461] acpiphp: Slot [28] registered > [ 0.151786] acpiphp: Slot [29] registered > [ 0.152104] acpiphp: Slot [30] registered > [ 0.152426] acpiphp: Slot [31] registered > [ 0.152741] PCI host bridge to bus 0000:00 > [ 0.153059] pci_bus 0000:00: root bus resource [bus 00-ff] > [ 0.153478] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] > [ 0.153991] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] > [ 0.154508] pci_bus 0000:00: root bus resource [mem > 0x000a0000-0x000bffff window] > [ 0.155072] pci_bus 0000:00: root bus resource [mem > 0x7d000000-0xfebfffff window] > [ 0.162550] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] > [ 0.163097] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] > [ 0.163590] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] > [ 0.164129] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] > [ 0.165004] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by > PIIX4 ACPI > [ 0.165564] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB > [ 0.223140] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) > [ 0.223712] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) > [ 0.224245] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) > [ 0.224789] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) > [ 0.225296] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) > [ 0.225817] ACPI: Enabled 2 GPEs in block 00 to 0F > [ 0.226262] vgaarb: loaded > [ 0.227000] SCSI subsystem initialized > [ 0.227314] ACPI: bus type USB registered > [ 0.227640] usbcore: registered new interface driver usbfs > [ 0.228068] usbcore: registered new interface driver hub > [ 0.228487] usbcore: registered new device driver usb > [ 0.228936] PCI: Using ACPI for IRQ routing > [ 0.229436] NetLabel: Initializing > [ 0.230112] NetLabel: domain hash size = 128 > [ 0.230455] NetLabel: protocols = UNLABELED CIPSOv4 > [ 0.230843] NetLabel: unlabeled traffic allowed by default > [ 0.231317] amd_nb: Cannot enumerate AMD northbridges > [ 0.231722] Switched to clocksource kvm-clock > [ 0.235503] pnp: PnP ACPI init > [ 0.235767] ACPI: bus type PNP registered > [ 0.236396] pnp: PnP ACPI: found 5 devices > [ 0.236716] ACPI: bus type PNP unregistered > [ 0.242333] NET: Registered protocol family 2 > [ 0.242806] TCP established hash table entries: 16384 (order: 5, > 131072 bytes) > [ 0.243384] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) > [ 0.243907] TCP: Hash tables configured (established 16384 bind 16384) > [ 0.244414] TCP: reno registered > [ 0.244668] UDP hash table entries: 1024 (order: 3, 32768 bytes) > [ 0.245130] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) > [ 0.245656] NET: Registered protocol family 1 > [ 0.246013] pci 0000:00:00.0: Limiting direct PCI/PCI transfers > [ 0.246473] pci 0000:00:01.0: PIIX3: Enabling Passive Release > [ 0.246924] pci 0000:00:01.0: Activating ISA DMA hang workarounds > [ 0.247457] Unpacking initramfs... > [ 0.249930] Freeing initrd memory: 3192k freed > [ 0.251174] sha1_ssse3: Using AVX optimized SHA-1 implementation > [ 0.251706] sha256_ssse3: Using AVX2 optimized SHA-256 implementation > [ 0.252355] futex hash table entries: 256 (order: 2, 16384 bytes) > [ 0.252836] Initialise system trusted keyring > [ 0.253187] audit: initializing netlink socket (disabled) > [ 0.253610] type=2000 audit(1516825598.479:1): initialized > [ 0.275426] HugeTLB registered 1 GB page size, pre-allocated 0 pages > [ 0.275927] HugeTLB registered 2 MB page size, pre-allocated 0 pages > [ 0.277129] zpool: loaded > [ 0.277350] zbud: loaded > [ 0.277669] VFS: Disk quotas dquot_6.5.2 > [ 0.277998] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) > [ 0.278609] msgmni has been set to 3901 > [ 0.278956] Key type big_key registered > [ 0.279450] NET: Registered protocol family 38 > [ 0.279810] Key type asymmetric registered > [ 0.280125] Asymmetric key parser 'x509' registered > [ 0.280523] Block layer SCSI generic (bsg) driver version 0.4 > loaded (major 250) > [ 0.281107] io scheduler noop registered > [ 0.281416] io scheduler deadline registered (default) > [ 0.281839] io scheduler cfq registered > [ 0.282216] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 > [ 0.282648] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 > [ 0.283250] input: Power Button as > /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 > [ 0.283835] ACPI: Power Button [PWRF] > [ 0.284207] GHES: HEST is not enabled! > [ 0.284534] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled > [ 0.307889] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A > [ 0.308457] Non-volatile memory driver v1.3 > [ 0.308809] Linux agpgart interface v0.103 > [ 0.309200] crash memory driver: version 1.1 > [ 0.309568] rdac: device handler registered > [ 0.309913] hp_sw: device handler registered > [ 0.310247] emc: device handler registered > [ 0.310565] alua: device handler registered > [ 0.310922] libphy: Fixed MDIO Bus: probed > [ 0.311267] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver > [ 0.311780] ehci-pci: EHCI PCI platform driver > [ 0.312129] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver > [ 0.312609] ohci-pci: OHCI PCI platform driver > [ 0.312958] uhci_hcd: USB Universal Host Controller Interface driver > [ 0.313474] usbcore: registered new interface driver usbserial > [ 0.313926] usbcore: registered new interface driver usbserial_generic > [ 0.314428] usbserial: USB Serial support registered for generic > [ 0.314911] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] > at 0x60,0x64 irq 1,12 > [ 0.316032] serio: i8042 KBD port at 0x60,0x64 irq 1 > [ 0.316418] serio: i8042 AUX port at 0x60,0x64 irq 12 > [ 0.316857] mousedev: PS/2 mouse device common for all mice > [ 0.317468] input: AT Translated Set 2 keyboard as > /devices/platform/i8042/serio0/input/input1 > [ 0.318561] input: VirtualPS/2 VMware VMMouse as > /devices/platform/i8042/serio1/input/input2 > [ 0.319363] input: VirtualPS/2 VMware VMMouse as > /devices/platform/i8042/serio1/input/input3 > [ 0.320042] rtc_cmos 00:00: RTC can wake from S4 > [ 0.320573] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 > [ 0.321099] rtc_cmos 00:00: alarms up to one day, y3k, 114 bytes nvram > [ 0.321632] cpuidle: using governor menu > [ 0.321989] hidraw: raw HID events driver (C) Jiri Kosina > [ 0.322467] usbcore: registered new interface driver usbhid > [ 0.322894] usbhid: USB HID core driver > [ 0.323734] drop_monitor: Initializing network drop monitor service > [ 0.324272] TCP: cubic registered > [ 0.324537] Initializing XFRM netlink socket > [ 0.324936] NET: Registered protocol family 10 > [ 0.325410] NET: Registered protocol family 17 > [ 0.325872] microcode: CPU0 sig=0x406f1, pf=0x1, revision=0x1 > [ 0.326331] microcode: Microcode Update Driver: v2.01 > <tigran(a)aivazian.fsnet.co.uk&gt;, Peter Oruba > [ 0.327060] Loading compiled-in X.509 certificates > [ 0.327855] Loaded X.509 cert 'Red Hat Enterprise Linux Driver > Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80' > [ 0.329151] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch > signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8' > [ 0.330379] Loaded X.509 cert 'Red Hat Enterprise Linux kernel > signing key: 34fc3b85a61b8fead6e9e905e7e602a1f7fa049a' > [ 0.331196] registered taskstats version 1 > [ 0.331639] Key type trusted registered > [ 0.332056] Key type encrypted registered > [ 0.332920] IMA: No TPM chip found, activating TPM-bypass! > [ 0.333605] Magic number: 2:270:448 > [ 0.333970] rtc_cmos 00:00: setting system clock to 2018-01-24 > 20:26:38 UTC (1516825598) > [ 0.335302] Freeing unused kernel memory: 1764k freed > [ 0.339427] alg: No test for crc32 (crc32-pclmul) > [ 0.342995] alg: No test for crc32 (crc32-generic) > [ 0.352535] scsi host0: ata_piix > [ 0.352853] scsi host1: ata_piix > [ 0.353127] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0c0 irq 14 > [ 0.353645] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0c8 irq 15 > [ 0.541003] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 > [ 0.545766] random: fast init done > [ 0.548737] random: crng init done > [ 0.565923] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 > [ 0.567592] scsi host2: Virtio SCSI HBA > [ 0.569801] scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK > 2.5+ PQ: 0 ANSI: 5 > [ 0.570526] scsi 2:0:1:0: Direct-Access QEMU QEMU HARDDISK > 2.5+ PQ: 0 ANSI: 5 > [ 0.580538] sd 2:0:0:0: [sda] 104857600 512-byte logical blocks: > (53.6 GB/50.0 GiB) > [ 0.581264] sd 2:0:1:0: [sdb] 8388608 512-byte logical blocks: > (4.29 GB/4.00 GiB) > [ 0.581894] sd 2:0:0:0: [sda] Write Protect is off > [ 0.582312] sd 2:0:0:0: [sda] Write cache: enabled, read cache: > enabled, doesn't support DPO or FUA > [ 0.583032] sd 2:0:1:0: [sdb] Write Protect is off > [ 0.583444] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: > enabled, doesn't support DPO or FUA > [ 0.586373] sd 2:0:1:0: [sdb] Attached SCSI disk > [ 0.602190] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 > [ 0.636655] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 > [ 1.253809] tsc: Refined TSC clocksource calibration: 2099.994 MHz > [ 1.510203] sda: sda1 sda2 sda3 > [ 1.511710] sd 2:0:0:0: [sda] Attached SCSI disk > [ 1.528245] EXT4-fs (sdb): mounting ext2 file system using the ext4 subsystem > [ 1.530353] EXT4-fs (sdb): mounted filesystem without journal. Opts: > [/usr/lib/tmpfiles.d/systemd.conf:11] Unknown group 'utmp'. > [/usr/lib/tmpfiles.d/systemd.conf:19] Unknown user 'systemd-network'. > [/usr/lib/tmpfiles.d/systemd.conf:20] Unknown user 'systemd-network'. > [/usr/lib/tmpfiles.d/systemd.conf:21] Unknown user 'systemd-network'. > [/usr/lib/tmpfiles.d/systemd.conf:25] Unknown group 'systemd-journal'. > [/usr/lib/tmpfiles.d/systemd.conf:26] Unknown group 'systemd-journal'. > [ 1.650422] input: PC Speaker as /devices/platform/pcspkr/input/input4 > [ 1.655216] piix4_smbus 0000:00:01.3: SMBus Host Controller at > 0x700, revision 0 > [ 1.694118] sd 2:0:0:0: Attached scsi generic sg0 type 0 > [ 1.696802] sd 2:0:1:0: Attached scsi generic sg1 type 0 > [ 1.698009] FDC 0 is a S82078B > [ 1.710807] AES CTR mode by8 optimization enabled > [ 1.724293] ppdev: user-space parallel port driver > [ 1.732252] Error: Driver 'pcspkr' is already registered, aborting... > [ 1.734673] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) > [ 1.746232] EDAC MC: Ver: 3.0.0 > [ 1.749324] EDAC sbridge: Ver: 1.1.1 > [ 25.658309] device-mapper: uevent: version 1.0.3 > [ 25.659092] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) > initialised: dm-devel(a)redhat.com > [ 57.8] Inspecting the overlay > [ 51.302190] EXT4-fs (sda1): mounted filesystem with ordered data > mode. Opts: (null) > [ 58.667082] EXT4-fs (dm-1): mounted filesystem with ordered data > mode. Opts: (null) > [ 61.147593] EXT4-fs (dm-4): mounted filesystem with ordered data > mode. Opts: (null) > [ 63.977572] EXT4-fs (dm-0): mounted filesystem with ordered data > mode. Opts: (null) > [ 75.614795] EXT4-fs (dm-6): mounted filesystem with ordered data > mode. Opts: (null) > [ 80.782266] EXT4-fs (dm-5): mounted filesystem with ordered data > mode. Opts: (null) > [ 98.734329] EXT4-fs (dm-2): mounted filesystem with ordered data > mode. Opts: (null) > [ 102.090148] EXT4-fs (dm-7): mounted filesystem with ordered data > mode. Opts: (null) > [ 105.057661] EXT4-fs (dm-3): mounted filesystem with ordered data > mode. Opts: (null) > [ 108.085788] EXT4-fs (dm-9): mounted filesystem with ordered data > mode. Opts: (null) > [ 111.328257] EXT4-fs (dm-8): mounted filesystem with ordered data > mode. Opts: (null) > [ 112.201934] EXT4-fs (dm-0): mounted filesystem with ordered data > mode. Opts: (null) > [ 112.212101] EXT4-fs (dm-2): mounted filesystem with ordered data > mode. Opts: (null) > [ 112.221689] EXT4-fs (dm-5): mounted filesystem with ordered data > mode. Opts: (null) > [ 112.233016] EXT4-fs (dm-6): mounted filesystem with ordered data > mode. Opts: (null) > [ 112.971075] EXT4-fs (dm-4): mounted filesystem with ordered data > mode. Opts: (null) > [ 113.788961] EXT4-fs (dm-1): mounted filesystem with ordered data > mode. Opts: (null) > [ 113.799156] EXT4-fs (sda1): mounted filesystem with ordered data > mode. Opts: (null) > [ 113.811402] EXT4-fs (dm-3): mounted filesystem with ordered data > mode. Opts: (null) > [ 113.823347] EXT4-fs (dm-9): mounted filesystem with ordered data > mode. Opts: (null) > [ 115.345857] EXT4-fs (dm-7): mounted filesystem with ordered data > mode. Opts: (null) > [ 115.356280] EXT4-fs (dm-8): mounted filesystem with ordered data > mode. Opts: (null) > [ 476.5] Checking for sufficient free disk space in the guest > [ 476.5] Estimating space required on target for each disk > [ 476.5] Converting Red Hat Enterprise Linux Server 7.4 (Maipo) to run on KVM > [ 1072.265252] dracut[1565] No '/dev/log' or 'logger' included for > syslog logging > [ 1076.444899] dracut[1565] Executing: /sbin/dracut --verbose > --add-drivers "virtio virtio_ring virtio_blk virtio_scsi virtio_net > virtio_pci" /boot/initramfs-3.10.0-693.el7.x86_64.img > 3.10.0-693.el7.x86_64 > [ 1104.118050] dracut[1565] dracut module 'busybox' will not be > installed, because command 'busybox' could not be found! > [ 1111.893587] dracut[1565] dracut module 'crypt' will not be > installed, because command 'cryptsetup' could not be found! > [ 1112.694542] dracut[1565] dracut module 'dmraid' will not be > installed, because command 'dmraid' could not be found! > [ 1117.763735] dracut[1565] dracut module 'mdraid' will not be > installed, because command 'mdadm' could not be found! > [ 1117.769004] dracut[1565] dracut module 'multipath' will not be > installed, because command 'multipath' could not be found! > [ 1122.366992] dracut[1565] dracut module 'cifs' will not be > installed, because command 'mount.cifs' could not be found! > [ 1122.387968] dracut[1565] dracut module 'iscsi' will not be > installed, because command 'iscsistart' could not be found! > [ 1122.390569] dracut[1565] dracut module 'iscsi' will not be > installed, because command 'iscsi-iname' could not be found! > [ 1140.889553] dracut[1565] dracut module 'busybox' will not be > installed, because command 'busybox' could not be found! > [ 1140.910458] dracut[1565] dracut module 'crypt' will not be > installed, because command 'cryptsetup' could not be found! > [ 1140.915646] dracut[1565] dracut module 'dmraid' will not be > installed, because command 'dmraid' could not be found! > [ 1140.924489] dracut[1565] dracut module 'mdraid' will not be > installed, because command 'mdadm' could not be found! > [ 1140.928995] dracut[1565] dracut module 'multipath' will not be > installed, because command 'multipath' could not be found! > [ 1140.939832] dracut[1565] dracut module 'cifs' will not be > installed, because command 'mount.cifs' could not be found! > [ 1140.954810] dracut[1565] dracut module 'iscsi' will not be > installed, because command 'iscsistart' could not be found! > [ 1140.957229] dracut[1565] dracut module 'iscsi' will not be > installed, because command 'iscsi-iname' could not be found! > [ 1142.066303] dracut[1565] *** Including module: bash *** > [ 1142.073837] dracut[1565] *** Including module: nss-softokn *** > [ 1143.838047] dracut[1565] *** Including module: i18n *** > [ 1230.935044] dracut[1565] *** Including module: network *** > [ 1323.749409] dracut[1565] *** Including module: ifcfg *** > [ 1323.755682] dracut[1565] *** Including module: drm *** > [ 1340.716219] dracut[1565] *** Including module: plymouth *** > [ 1359.941093] dracut[1565] *** Including module: dm *** > [ 1366.392221] dracut[1565] Skipping udev rule: 64-device-mapper.rules > [ 1366.394670] dracut[1565] Skipping udev rule: 60-persistent-storage-dm.rules > [ 1366.397021] dracut[1565] Skipping udev rule: 55-dm.rules > [ 1375.796931] dracut[1565] *** Including module: kernel-modules *** > [ 1627.998656] dracut[1565] *** Including module: lvm *** > [ 1631.138460] dracut[1565] Skipping udev rule: 64-device-mapper.rules > [ 1631.141015] dracut[1565] Skipping udev rule: 56-lvm.rules > [ 1631.143409] dracut[1565] Skipping udev rule: 60-persistent-storage-lvm.rules > [ 1635.270706] dracut[1565] *** Including module: qemu *** > [ 1635.277842] dracut[1565] *** Including module: rootfs-block *** > [ 1636.845616] dracut[1565] *** Including module: terminfo *** > [ 1639.189294] dracut[1565] *** Including module: udev-rules *** > [ 1640.076624] dracut[1565] Skipping udev rule: 40-redhat-cpu-hotplug.rules > [ 1649.962889] dracut[1565] Skipping udev rule: 91-permissions.rules > [ 1651.008527] dracut[1565] *** Including module: biosdevname *** > [ 1651.921630] dracut[1565] *** Including module: systemd *** > [ 1685.124521] dracut[1565] *** Including module: usrmount *** > [ 1685.128532] dracut[1565] *** Including module: base *** > [ 1694.743507] dracut[1565] *** Including module: fs-lib *** > [ 1696.295216] dracut[1565] *** Including module: shutdown *** > [ 1698.578228] dracut[1565] *** Including modules done *** > [ 1699.586287] dracut[1565] *** Installing kernel module dependencies > and firmware *** > [ 1717.505952] dracut[1565] *** Installing kernel module dependencies > and firmware done *** > [ 1724.539224] dracut[1565] *** Resolving executable dependencies *** > [ 1844.709874] dracut[1565] *** Resolving executable dependencies done*** > [ 1844.723313] dracut[1565] *** Hardlinking files *** > [ 1847.281611] dracut[1565] *** Hardlinking files done *** > [ 1847.284119] dracut[1565] *** Stripping files *** > [ 1908.635888] dracut[1565] *** Stripping files done *** > [ 1908.638262] dracut[1565] *** Generating early-microcode cpio image > contents *** > [ 1908.645054] dracut[1565] *** Constructing GenuineIntel.bin **** > [ 1909.567397] dracut[1565] *** Store current command line parameters *** > [ 1909.571686] dracut[1565] *** Creating image file *** > [ 1909.574239] dracut[1565] *** Creating microcode section *** > [ 1911.789907] dracut[1565] *** Created microcode section *** > [ 1921.680575] dracut[1565] *** Creating image file done *** > [ 1926.764407] dracut[1565] *** Creating initramfs image file > '/boot/initramfs-3.10.0-693.el7.x86_64.img' done *** > [1994.1] Mapping filesystem data to avoid copying unused and blank areas > [ 1984.841231] EXT4-fs (dm-8): mounted filesystem with ordered data > mode. Opts: discard > [ 1987.252106] EXT4-fs (dm-9): mounted filesystem with ordered data > mode. Opts: discard > [ 1990.531305] EXT4-fs (dm-3): mounted filesystem with ordered data > mode. Opts: discard > [ 1992.903109] EXT4-fs (dm-7): mounted filesystem with ordered data > mode. Opts: discard > [ 1995.876230] EXT4-fs (dm-2): mounted filesystem with ordered data > mode. Opts: discard > [ 1995.986384] EXT4-fs (dm-5): mounted filesystem with ordered data > mode. Opts: discard > [ 1997.748087] EXT4-fs (dm-6): mounted filesystem with ordered data > mode. Opts: discard > [ 1997.785914] EXT4-fs (dm-0): mounted filesystem with ordered data > mode. Opts: discard > [ 1997.824003] EXT4-fs (dm-4): mounted filesystem with ordered data > mode. Opts: discard > [ 2000.172658] EXT4-fs (dm-1): mounted filesystem with ordered data > mode. Opts: discard > [ 2001.214202] EXT4-fs (sda1): mounted filesystem with ordered data > mode. Opts: discard > [2010.7] Closing the overlay > [2010.7] Checking if the guest needs BIOS or UEFI to boot > [2010.7] Assigning disks to buses > [2010.7] Copying disk 1/1 to > /rhev/data-center/e8263fb4-114d-4706-b1c0-5defcd15d16b/a118578a-4cf2-4e0c-ac47-20e9f0321da1/images/1a93e503-ce57-4631-8dd2-eeeae45866ca/88d92582-0f53-43b0-89ff-af1c17ea8618 > (raw) > [7000.4] Creating output metadata > [7000.4] Finishing off > > Any help is appreciated. > > Luca > > > > -- > "E' assurdo impiegare gli uomini di intelligenza eccellente per fare > calcoli che potrebbero essere affidati a chiunque se si usassero delle > macchine" > Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) > > "Internet è la più grande biblioteca del mondo. > Ma il problema è che i libri sono tutti sparsi sul pavimento" > John Allen Paulos, Matematico (1945-vivente) > > Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <lorenzetto.luca(a)gmail.com&gt; > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

On Thu, Jan 25, 2018 at 10:08 AM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: > There's got to be some difference between your staging environment and > your production environment, and I'm pretty sure it has nothing to do > with the version of oVirt. > > Are you running virt-v2v inside a virtual machine, and previously you > ran it on bare-metal? Or did you disable nested KVM? That seems like > the most likely explanation for the difference (although I'm surprised > that the difference is so large). > > Rich. > Hello Rich, i'm running virt-v2v throught the import option of oVirt.

[root@kvm01 ~]# rpm -qa virt-v2v virt-v2v-1.36.3-6.el7_4.3.x86_64 [root@kvm01 ~]# rpm -qa libguestfs libguestfs-1.36.3-6.el7_4.3.x86_64 [root@kvm01 ~]# rpm -qa "redhat-virtualization-host-image-update*" redhat-virtualization-host-image-update-placeholder-4.1-8.1.el7.noarch redhat-virtualization-host-image-update-4.1-20171207.0.el7_4.noarch (yes, i'm running RHV, but i think this shouldn't change the behaviour) I don't set anything in the commandline or whatever, i set only the source and destination throught the API. So virt-v2v is coordinated via vdsm and runs on the bare-metal host. The network distance is "0", because vcenter, source vmware hosts, kvm hosts and ovirt hosts lies in the same network. The only annotation is that also vCenter is a VM, running on esx environment. Network interfaces both on source and destination are 10Gbit, but there may be a little slowdown on vcenter side because has to get the data from esx's datastore and forward to the ovirt host.

Hello Richard, unfortunately upgrading virt-v2v is not an option. Would be nice, but integration with vdsm is not yet ready for that options. On Thu, Jan 25, 2018 at 11:06 AM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: [cut] > I don't know why it slowed down, but I'm pretty sure it's got nothing > to do with the version of oVirt/RHV. Especially in the initial phase > where it's virt-v2v reading the guest from vCenter. Something must > have changed or be different in the test and production environments. > > Are you converting the same guests? virt-v2v is data-driven, so > different guests require different operations, and those can take > different amount of time to run. > I'm not migrating the same guests, i'm migrating different guest, but most of them share the same os baseline. Most of these vms are from the same RHEL 7 template and have little data difference (few gigs). Do you know which is the performance impact on vcenter? I'd like to tune as best as possible the vcenter to improve the migration time.

We have to migrate ~300 guests, and our maintenance window is very short. We don't want continue the migration for months.

On Fri, Feb 2, 2018 at 12:52 PM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote: > There is a section about this in the virt-v2v man page. I'm on > a train at the moment but you should be able to find it. Try to > run many conversions, at least 4 or 8 would be good places to start. Hello Richard, read the man but found nothing explicit about resource usage. Anyway, digging on our setup i found out that vcenter when on low cpu usage is 95%. I think our windows admins should take care of this.

http://libguestfs.org/virt-v2v.1.html#vmware-vcenter-resources You should be able to run multiple conversions in parallel to improve throughput. The only long-term solution is to use a different method such as VMX over SSH. vCenter is just fundamentally bad.

Il 6 feb 2018 10:52 AM, "Yaniv Kaul" <ykaul(a)redhat.com&gt; ha scritto: I assume its network interfaces are also a bottleneck as well. Certainly if they are 1g. Y. That's not the case, vcenter uses 10g and also all the involved hosts. We first supposed the culprit was network, but investigations has cleared its position. Network usage is under 40% with 4 ongoing migrations.