1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
|
boot: 0.975
network: 0.973
device: 0.972
instruction: 0.972
other: 0.971
assembly: 0.970
semantic: 0.969
socket: 0.969
graphic: 0.968
KVM: 0.968
vnc: 0.958
mistranslation: 0.926
PowerPC L2(nested virt) kvm guest fails to boot with ic-mode=dual,kernel-irqchip=on - `KVM is too old to support ic-mode=dual,kernel-irqchip=on`
Env:
HW: Power 9 DD2.3
Host L0: 5.8.0-rc5-g8ba4ffcd8
Qemu: 5.0.50 (v5.0.0-533-gdebe78ce14)
Libvirt: 6.4.0
L1: 5.8.0-rc5-ge9919e11e
qemu_version': '5.0.50 (v5.1.0-rc2-dirty)
libvirt_version': '6.4.0'
L2: 5.8.0-rc7-g6ba1b005f
1. boot a L2 KVM guest with `ic-mode=dual,kernel-irqchip=on`
/usr/bin/virt-install --connect=qemu:///system --hvm --accelerate --name 'vm1' --machine pseries --memory=8192 --vcpu=8,maxvcpus=8,sockets=1,cores=2,t
hreads=4 --import --nographics --serial pty --memballoon model=virtio --disk path=/home/tests/data/avocado-vt/images/f31-ppc64le.qcow2,bus=virtio,size=10,format=qcow2 --network
=bridge=virbr0,model=virtio,mac=52:54:00:e6:fe:f6 --mac=52:54:00:e6:fe:f6 --boot emulator=/usr/share/avocado-plugins-vt/bin/qemu,kernel=/tmp/linux/vmlinux,kernel_args="root=/de
v/vda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0" --noautoconsole --qemu-commandline=" -M pseries,ic-mode=dual,kernel-irqchip=on"
ERROR internal error: process exited while connecting to monitor: 2020-08-04T11:12:53.304482Z qemu: KVM is too old to support ic-mode=dual,kernel-irqchip=on
Qemu Log:
```
/usr/share/avocado-plugins-vt/bin/qemu \
-name guest=vm1,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-vm1/master-key.aes \
-machine pseries-5.1,accel=kvm,usb=off,dump-guest-core=off \
-cpu POWER9 \
-m 8192 \
-overcommit mem-lock=off \
-smp 8,sockets=1,dies=1,cores=2,threads=4 \
-uuid 20a3351b-2776-4e75-9059-c070fe3dd44b \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=34,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-kernel /tmp/linux/vmlinux \
-append 'root=/dev/vda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0' \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 \
-blockdev '{"driver":"file","filename":"/home/tests/data/avocado-vt/images/f31-ppc64le.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk0,bootindex=1 \
-netdev tap,fd=37,id=hostnet0,vhost=on,vhostfd=38 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:e6:fe:f6,bus=pci.0,addr=0x1 \
-chardev pty,id=charserial0 \
-device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 \
-chardev socket,id=charchannel0,fd=39,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \
-M pseries,ic-mode=dual,kernel-irqchip=on \
-msg timestamp=on
2020-08-04 11:12:53.169+0000: Domain id=5 is tainted: custom-argv
2020-08-04 11:12:53.179+0000: 11120: info : libvirt version: 6.4.0, package: 1.fc31 (Unknown, 2020-06-02-05:09:40, ltc-wspoon4.aus.stglabs.ibm.com)
2020-08-04 11:12:53.179+0000: 11120: info : hostname: atest-guest
2020-08-04 11:12:53.179+0000: 11120: info : virObjectUnref:347 : OBJECT_UNREF: obj=0x7fff0c117c40
char device redirected to /dev/pts/0 (label charserial0)
2020-08-04T11:12:53.304482Z qemu: KVM is too old to support ic-mode=dual,kernel-irqchip=on
2020-08-04 11:12:53.694+0000: shutting down, reason=failed
```
This is currently expected because the L2 KVM guest uses the historical KVM XICS device (not the XICS-on-XIVE one) and this can be only created once during the VM lifetime for the moment.
This is a limitation in KVM, that can be addressed in several ways:
1) change the historical KVM XICS device to implement the release() method instead of destroy(), so that userspace can close() and re-create the device multiple times during the VM lifetime, as we have already done in KVM XIVE and KVM XICS-on-XIVE for powernv
2) have the KVM XICS-on-XIVE device to work under pseries
I already have a tentative patch for 1) and I guess 2) would be part of a more global work to supporting nested KVM XIVE.
But it is definitely not an issue in QEMU.
As per this the table https://www.qemu.org/docs/master/specs/ppc-spapr-xive.html#kvm-negotiation
reported qemu error msg "KVM is too old to support ic-mode=dual,kernel-irqchip=on" indicates the
guest os is legacy, but that's not the case here, whereas kernel levels are near upstream which has support for xive.
My understanding of the env I used as below
Level | XIVE KVM support | XIVE support(in kernel or emulation)
--------------------------------------------------
L0 | Yes | Yes
L1 | No | Yes(booted with irqchip: in-kernel)
L2 | No | Yes
So, ideally when a L2 guest is started with ic-mode=dual,kernel-irqchip=on, we should have seen below error
(2) QEMU fails with ``kernel_irqchip requested but unavailable:
IRQ_XIVE capability must be present for KVM``
but we actually saw the reported one, which is misleading.
this section of table in particular, https://www.qemu.org/docs/master/specs/ppc-spapr-xive.html#no-xive-support-in-kvm
Hmm... the documentation might need an update. I'll have a look.
Posted a patch to the list.
http://patchwork.ozlabs.org/project<email address hidden>/
Satheesh,
Can you please review and test ?
@Greg,
Thanks for the patch, I see it already got applied into 5.2, tested and works fine,
# git log -2 --oneline
1972794 (HEAD -> master) spapr: Clarify error and documentation for broken KVM XICS
e1d322c (grafted, tag: v5.1.0-rc3, origin/master, origin/HEAD) Update version for v5.1.0-rc3 release
# /usr/bin/virt-install --connect=qemu:///system --hvm --accelerate --name 'vm1' --machine pseries --memory=8192 --vcpu=8,maxvcpus=8,sockets=1,cores=8,threads=1 --import --nographics --serial pty --memballoon model=virtio --controller type=scsi,model=virtio-scsi --disk path=/home/tests/data/avocado-vt/images/f31-ppc64le.qcow2,bus=scsi,size=10,format=qcow2 --network=bridge=virbr0,model=virtio,mac=52:54:00:5c:f1:fe --mac=52:54:00:5c:f1:fe --boot emulator=/home/qemu/ppc64-softmmu/qemu-system-ppc64,kernel=/boot/vmlinuz-5.8.0-rc5-ge9919e11e,kernel_args="root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0" --noautoconsole --qemu-commandline=" -M pseries,ic-mode=dual,kernel-irqchip=on";virsh console vm1
Starting install...
ERROR internal error: process exited while connecting to monitor: 2020-08-07T07:05:38.633057Z qemu-system-ppc64: KVM is incompatible with ic-mode=dual,kernel-irqchip=on
This can happen with an old KVM or in a KVM nested guest.
Try without kernel-irqchip or with kernel-irqchip=off.
Regards,
-Satheesh
Released with QEMU v5.2.0.
|