path: root/results/scraper/launchpad-without-comments/1797332

qemu nested virtualization is not working with Ubuntu16.04 + Intel CPU

# 1 What am I trying to do ? #

I want to use `libvirt` `qemu/KVM` with **nested virtualization** like described
in [1] and [2].
**But it does not work with Ubuntu16.04.** It worked some times ago, but not
anymore.


I want 2 levels of virtualization like this:

* L0 – the bare metal host, running KVM on `Ubuntu 16.04`
* L1 – a `Ubuntu 16.04` VM running on L0; also called the "guest hypervisor" 
  — as it itself is capable of running KVM
* L2 – a `Ubuntu 16.04` VM running on L1, also called the "nested guest"


[1] https://docs.fedoraproject.org/en-US/quick-docs/using-nested-virtualization-in-kvm/
[2] https://www.linux-kvm.org/page/Nested_Guests


My goal is to deploy an `OpenStack` environnement on top of VMs rather than on
bare metal hosts for convenience for a lab experiment. As a result, the 
`OpenStack` nodes are L1 VMs. Compute nodes are L1 VMs as well and the VMs 
created with `OpenStack` and wich are running on the compute nodes are L2 VMs.






# 2 What is my problem ? #

I can **not** run my 2nd levels of virtualization in 16.04: 

* L0 is just fine: running `Ubuntu 16.04.5 LTS`, installed with the `.iso` image
* L1: I install `libvirt` + `KVM` on L0. I can run VMs like the `Ubuntu16.04` 
  cloud image on L0.
* L2: I install `libvirt` + `KVM` on L1 as well. But I **can not** run VMs on 
  L1: I get `kernel panic` or `general protection fault`.


**But if I do the same with Ubuntu18.04** (on the same hardware) instead of 
`Ubuntu16.04`, it works without faults.
I don't change the configuration or `virt-install scripts` (other than using 
the 18.04 .iso and cloud image).






# 3 My libvirt installation for Ubuntu16.04 #

I install `libvir KVM` in both L0 and L1 using a custom repository [3] from 
`OpenStack` team, because their version of libvirt in this repo is newer than  
the one on Ubuntu 16.04 official repo and it match the version of `libvirt` 
in Ubuntu 18.04.

[3] https://wiki.ubuntu.com/OpenStack/CloudArchive






# 4 hardware and CPU #

CPU is:
> Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
> Intel virt is enable in the bios/uefi.

The rest is standard HDD, standard I/O...






# 5 .iso and cloud image #

I download .iso for L0 bare metal server and cloud image 
for L1/L2 VMs from official repository:

Ubuntu 16.04
 * http://releases.ubuntu.com/16.04/
 * https://cloud-images.ubuntu.com/releases/16.04/release/

Ubuntu 18.04
 * http://releases.ubuntu.com/bionic/
 * https://cloud-images.ubuntu.com/releases/18.04/release/






# 6 Details #

## Details about L0 Ubuntu 16.04 bare metal host ##
L0 is running `Ubuntu 16.04.5 LTS` installed with the .iso.


**kernel**
```
user@L0:~$ uname -a
Linux L0 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
```

**libvirt version** running on L0
```
user@L0:~$ virsh version
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.11.1
```

**qemu version detail**
```
ukvm2@kvm2:~$ qemu-system-x86_64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.5~cloud0)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
```

**KVM acceleration**
```
user@L0:~$ kvm-ok 
INFO: /dev/kvm exists
KVM acceleration can be used
```

**nested parameter**
```
user@L0:~$ cat /sys/module/kvm_intel/parameters/nested
Y
```

**number of CPU**
```
user@L0:~$ egrep -c '(vmx|svm)' /proc/cpuinfo
48
```



## Details about a L1 Ubuntu 16.04 VM ##
A VM in L1 (which is running on L0) which is running `Ubuntu 16.04.5 LTS` 
installed by a cloud image.

**kernel**
```
user@L1-VM:~$ uname -a
Linux L1 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
```

**libvirt version** running on the L1 VM
```
user@L1-VM:~$ sudo virsh version
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.11.1
```

**qemu version detail**
```
user@L1-VM:~$ qemu-system-x86_64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.5~cloud0)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
```

**KVM acceleration**
```
user@L1-VM:~$ kvm-ok 
INFO: /dev/kvm exists
KVM acceleration can be used
```

**nested parameter**
```
user@L1-VM:~$ cat /sys/module/kvm_intel/parameters/nested
Y
```

**number of CPU**, which are vCPU given by L0 to the L1 VM
I give 20 vCPU.
```
user@L1-VM:~$ egrep -c '(vmx|svm)' /proc/cpuinfo
20
```



## L1 VM virt-install script parameter ##
If you want to reproduce an L1 VM, I followed this [4]:

```
virt-install \
    --connect=qemu:///system \
    --name $VMName \
    --memory $RAM \
    --vcpus $VCPUS \
    --cpu host \
    --metadata description=$DESCRIPTION \
    --os-type linux \
    --os-variant ubuntu16.04 \
    --disk $DISK_PATH/$VMName.$DISK_FORMAT,size=$DISK_SIZE,bus=virtio \
    --disk $CFGIMG_PATH/config_$VMName.$DISK_FORMAT,device=cdrom \
    --network bridge=virbr0 \
    --graphics none \
    --console pty,target_type=serial \
    --hvm
```

[4] https://youth2009.org/post/kvm-with-ubuntu-cloud-image/



## Details about a L2 VM ##

I want to create a L2 `Ubuntu 16.04.5 LTS` VM installed by a cloud image VM 
within my L1 `KVM` VM. But whatever I do, my L2 VM crash before finishing to be 
instantiated. I get `kernel panic` or `general protection fault`.


Here is the log of an L2 VM after the instanciation failed:
```
user@L1-VM:~$ less /var/log/libvirt/qemu/VMNAME.log

2018-10-11T07:40:45.837151Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/1 (label charserial0)
2018-10-11T07:40:45.844279Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
2018-10-11T07:40:45.848532Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
```


If you want to reproduce an L2 VM running on L1, follow [4].


**However** a Cirros OS image can run on a L1 VM !






# 7 Thoughts #
I think this is a bug in either `Ubuntu16.04` or `libvirt`.
All the information are here to reproduce the bug, I think.


If I do the same with `Ubuntu 18.04`, on the same hardware, following the same 
steps but with Ubuntu 18.04 .iso and cloud image, it works.

It works if:

* L0 = Ubuntu18.04 (.iso) + qemu/KVM
* L1 = Ubuntu18.04 (cloud image) + qemu/KVM
* L2 = Ubuntu18.04 (cloud image)


It also works if:

* L0 = Ubuntu18.04 (.iso) + qemu/KVM
* L1 = Ubuntu18.04 (cloud image) + qemu/KVM
* L2 = Ubuntu16.04 (cloud image)




Thank you for your time reading !
--
nico