KVM: 0.860 other: 0.851 vnc: 0.827 device: 0.769 graphic: 0.739 performance: 0.739 permissions: 0.730 semantic: 0.727 boot: 0.690 debug: 0.689 socket: 0.655 files: 0.632 network: 0.629 PID: 0.621 VM failed to start in nested virtualization with error "KVM: entry failed, hardware error 0x0" Hi, I have 3 ubuntu nodes provisioned by IaaS. Then I tried to launch VM again in my ubuntu nodes. It's a little strange that VM could be started successfully in two nodes. And always failed in one nodes with error "KVM: entry failed, hardware error 0x0". When using virsh to resume the VM, it failed with following error, virsh # list Id Name State ---------------------------------- 1 default_vm-cirros paused virsh # resume default_vm-cirros error: Failed to resume domain default_vm-cirros error: internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required The detailed log from /var/log/libvirt/qemu/default_vm-cirros.log is as below. ``` 2019-06-18 09:55:52.397+0000: starting up libvirt version: 5.0.0, package: 1.fc28 (Unknown, 2019-01-22-08:04:34, 64723eea657e48d296e6beb0b1be9c4c), qemu version: 3.1.0qemu-3.1.0-4.fc28, kernel: 4.15.0-47-generic, hostname: vm-cirros LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ HOME=/root \ QEMU_AUDIO_DRV=none \ /usr/bin/qemu-system-x86_64 \ -name guest=default_vm-cirros,debug-threads=on \ -S \ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-default_vm-cirros/master-key.aes \ -machine pc-q35-3.1,accel=kvm,usb=off,dump-guest-core=off \ -cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc_adjust=on,mpx=on,avx512f=on,avx512cd=on,ssbd=on,xsaveopt=on,abm=on,invpcid=off \ -m 489 \ -realtime mlock=off \ -smp 1,sockets=1,cores=1,threads=1 \ -object iothread,id=iothread1 \ -uuid 0d2a2043-41c0-59c3-9b17-025022203668 \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=22,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \ -drive file=/var/run/kubevirt-ephemeral-disks/container-disk-data/default/vm-cirros/disk_containerdisk/disk-image.raw,format=raw,if=none,id=drive-ua-containerdisk,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.3,addr=0x0,drive=drive-ua-containerdisk,id=ua-containerdisk,bootindex=1,write-cache=on \ -drive file=/var/run/kubevirt-ephemeral-disks/cloud-init-data/default/vm-cirros/noCloud.iso,format=raw,if=none,id=drive-ua-cloudinitdisk,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=drive-ua-cloudinitdisk,id=ua-cloudinitdisk,write-cache=on \ -netdev tap,fd=24,id=hostua-default,vhost=on,vhostfd=25 \ -device virtio-net-pci,host_mtu=1430,netdev=hostua-default,id=ua-default,mac=16:57:38:cd:57:cb,bus=pci.1,addr=0x0 \ -chardev socket,id=charserial0,fd=26,server,nowait \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=27,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -vnc vnc=unix:/var/run/kubevirt-private/3b22a138-91af-11e9-af36-0016ac101123/virt-vnc \ -device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on KVM: entry failed, hardware error 0x0 EAX=00000000 EBX=00000000 ECX=00000000 EDX=000306d2 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 0000ffff 00009300 CS =f000 ffff0000 0000ffff 00009b00 SS =0000 00000000 0000ffff 00009300 DS =0000 00000000 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000 00000000 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 00000000 0000ffff IDT= 00000000 0000ffff CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=06 66 05 00 00 01 00 8e c1 26 66 a3 14 f5 66 5b 66 5e 66 c3 5b e0 00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ``` Ubuntu node version as follow, cat /etc/os-release NAME="Ubuntu" VERSION="18.04.2 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.2 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic Output of `uname -a` is: 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Any additional information needed, please let me know. Thx. The QEMU project is currently considering to move its bug tracking to another system. For this we need to know which bugs are still valid and which could be closed already. Thus we are setting older bugs to "Incomplete" now. If you still think this bug report here is valid, then please switch the state back to "New" within the next 60 days, otherwise this report will be marked as "Expired". Or please mark it as "Fix Released" if the problem has been solved with a newer version of QEMU already. Thank you and sorry for the inconvenience. [Expired for QEMU because there has been no activity for 60 days.]