1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
|
network: 0.943
other: 0.928
assembly: 0.928
device: 0.927
KVM: 0.909
socket: 0.905
instruction: 0.890
semantic: 0.885
vnc: 0.879
graphic: 0.866
mistranslation: 0.845
boot: 0.788
virsh start, after virsh managed save hangs and vm goes to paused state with qemu version v2.12.0-813-g5a5c383b13-dirty on powerpc
Host Env:
IBM Power8 with Fedora28 base with compiled upstream kernel, qemu, libvirt.
Host Kernel: 4.17.0-rc5-00069-g3acf4e395260
qemu-kvm(5a5c383b1373aeb6c87a0d6060f6c3dc7c53082b): v2.12.0-813-g5a5c383b13-dirty
libvirt(4804a4db33a37f828d033733bc47f6eff5d262c3):
Guest Kernel: 4.17.0-rc7
Steps to recreate:
Define a guest attached with above setup and start.
# virsh start avocado-vt-vm1
guest console;...
# uname -r
4.17.0-rc7
[root@atest-guest ~]# lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 3
On-line CPU(s) list: 0-2
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 3
NUMA node(s): 1
Model: 2.1 (pvr 004b 0201)
Model name: POWER8 (architected), altivec supported
Hypervisor vendor: KVM
Virtualization type: para
L1d cache: 64K
L1i cache: 32K
NUMA node0 CPU(s): 0-2
# virsh managedsave avocado-vt-vm1
Domain avocado-vt-vm1 state saved by libvirt
# virsh list
Id Name State
----------------------------------------------------
# virsh start avocado-vt-vm1 ----Hangs forever and vm state goes to paused.
# virsh list
Id Name State
----------------------------------------------------
87 avocado-vt-vm1 paused
P:S:- with same above setup, just changing the qemu-kvm comes bydefault with F28 works fine.
/usr/bin/qemu-kvm --version
QEMU emulator version 2.11.1(qemu-2.11.1-2.fc28)
Summary: with above other setup.
machine type pseries-2.12 and qemu-2.11.1-2.fc28 -Works fine.
machine type pseries-2.12/pseries-2.13 and qemu 5a5c383b1373aeb6c87a0d6060f6c3dc7c53082b - Does not work.
To recover from the failed state it requires below steps to be run.
# virsh destroy avocado-vt-vm1
Domain avocado-vt-vm1 destroyed
# virsh undefine --managed-save avocado-vt-vm1
Domain avocado-vt-vm1 has been undefined
On Mon, May 28, 2018 at 09:12:21AM -0000, Satheesh Rajendran wrote:
> Public bug reported:
>
> Host Env:
> IBM Power8 with Fedora28 base with compiled upstream kernel, qemu, libvirt.
>
> Host Kernel: 4.17.0-rc5-00069-g3acf4e395260
>
> qemu-kvm(5a5c383b1373aeb6c87a0d6060f6c3dc7c53082b):
> v2.12.0-813-g5a5c383b13-dirty
>
> libvirt(4804a4db33a37f828d033733bc47f6eff5d262c3):
>
> Guest Kernel: 4.17.0-rc7
>
> Steps to recreate:
> Define a guest attached with above setup and start.
> # virsh start avocado-vt-vm1
>
> guest console;...
> # uname -r
> 4.17.0-rc7
> [root@atest-guest ~]# lscpu
> Architecture: ppc64le
> Byte Order: Little Endian
> CPU(s): 3
> On-line CPU(s) list: 0-2
> Thread(s) per core: 1
> Core(s) per socket: 1
> Socket(s): 3
> NUMA node(s): 1
> Model: 2.1 (pvr 004b 0201)
> Model name: POWER8 (architected), altivec supported
> Hypervisor vendor: KVM
> Virtualization type: para
> L1d cache: 64K
> L1i cache: 32K
> NUMA node0 CPU(s): 0-2
>
>
> # virsh managedsave avocado-vt-vm1
>
> Domain avocado-vt-vm1 state saved by libvirt
>
> # virsh list
> Id Name State
> ----------------------------------------------------
>
> # virsh start avocado-vt-vm1 ----Hangs forever and vm state goes to
> paused.
Libvirt is using fd migration, right? If so, I suspect this is the
same problem with the iotest failure, and the fix should be in a pull
request:
Message-Id: <email address hidden>
Subject: [Qemu-devel] [PULL 1/2] migration: fix exec/fd migrations
Regards,
--
Peter Xu
with above patch compiled on top of latest upstream fails with below error:
# virsh managedsave avocado-vt-vm1
error: Failed to save domain avocado-vt-vm1 state
error: internal error: guest unexpectedly quit
rest of the behaviour same..
# virsh start avocado-vt-vm1 ----gets hung
---crtl+c --> to comeback to prompt
#
# virsh destroy avocado-vt-vm1
Domain avocado-vt-vm1 destroyed
# virsh undefine --managed-save avocado-vt-vm1
Domain avocado-vt-vm1 has been undefined
followed by further attempts saves the domains as reported but issue still same.
#virsh managedsave avocado-vt-vm1
Domain avocado-vt-vm1 state saved by libvirt
# virsh start avocado-vt-vm1 ----hung
# virsh list --all
Id Name State
----------------------------------------------------
98 avocado-vt-vm1 paused
I tried restarting libvirt, after which guest goes to shutoff state, with reason as crash in the qemu log
# service libvirtd restart
Redirecting to /bin/systemctl restart libvirtd.service
# virsh list --all
Id Name State
----------------------------------------------------
- avocado-vt-vm1 shut off
2018-05-28 12:59:46.748+0000: starting up libvirt version: 4.4.0, package: 1.fc28 (Unknown, 2018-05-28-03:15:39, 9.40.192.86), qemu version: 2.12.50v2.12.0-813-g5a5c383b13-dirty, kernel: 4.17.0-rc5-00069-g3acf4e395260, hostname: 9.40.192.86
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/share/avocado-plugins-vt/bin/qemu -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-97-avocado-vt-vm1/master-key.aes -machine pseries-2.13,accel=kvm,usb=off,dump-guest-core=off -m 1024 -realtime mlock=off -smp 2,maxcpus=4,sockets=4,cores=1,threads=1 -uuid ba3012d5-3244-47d9-bedc-0b60821f7cd1 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-97-avocado-vt-vm1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -kernel /home/kvmci/linux/vmlinux -append 'root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug' -device qemu-xhci,id=usb,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/avocado/data/avocado-vt/images/jeos-27-ppc64le.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3d:3e:3f,bus=pci.0,addr=0x1 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/avocado-vt-vm1-guest.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -sandbox off -msg timestamp=on
2018-05-28T12:59:46.826738Z qemu: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
2018-05-28 13:00:52.948+0000: shutting down, reason=saved
2018-05-28T13:00:52.950802Z qemu: terminating on signal 15 from pid 41456 (/usr/sbin/libvirtd)
2018-05-28 13:01:00.467+0000: starting up libvirt version: 4.4.0, package: 1.fc28 (Unknown, 2018-05-28-03:15:39, 9.40.192.86), qemu version: 2.12.50v2.12.0-813-g5a5c383b13-dirty, kernel: 4.17.0-rc5-00069-g3acf4e395260, hostname: 9.40.192.86
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/share/avocado-plugins-vt/bin/qemu -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-98-avocado-vt-vm1/master-key.aes -machine pseries-2.13,accel=kvm,usb=off,dump-guest-core=off -m 1024 -realtime mlock=off -smp 2,maxcpus=4,sockets=4,cores=1,threads=1 -uuid ba3012d5-3244-47d9-bedc-0b60821f7cd1 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-98-avocado-vt-vm1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -kernel /home/kvmci/linux/vmlinux -append 'root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug' -device qemu-xhci,id=usb,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/avocado/data/avocado-vt/images/jeos-27-ppc64le.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=31,id=hostnet0,vhost=on,vhostfd=33 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3d:3e:3f,bus=pci.0,addr=0x1 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/avocado-vt-vm1-guest.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -sandbox off -msg timestamp=on
2018-05-28T13:01:00.546872Z qemu: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
2018-05-28 13:02:56.434+0000: shutting down, reason=crashed <============
I was able to reproduce this with:
qemu - v2.12.0-813-g5a5c383
host/guest kernel - 4.11
libvirt - 3.9.0
It bisects to:
36c2f8b migration: Delay start of migration main routines
However, the issue did *not* reproduce with:
qemu - v2.12.0-865-ge609fa7
host/guest kernel - 4.11
libvirt - 3.9.0
As Peter suggested, it is fixed by:
0efc914 migration: fix exec/fd migrations
So perhaps there is still something on libvirt side? I'll try again with a more
recent one.
Could not reproduce with:
qemu - v2.12.0-865-ge609fa7
host/guest kernel - 4.11
libvirt - 4.4.0
and
qemu - v2.12.0-865-ge609fa7
host kernel - v4.17-rc7-22-g3d661e2
guest kernel - 4.11
libvirt - 4.4.0
So I'd say that this is fixed by:
https://git.qemu.org/?p=qemu.git;a=commitdiff;h=0efc914
Yes, tested again with below levels and not issue is not reproducible.
Issue is fixed!
qemu: 2.12.50 (v2.12.0-949-g392fba9f58-dirty)
host/guest kernel: 4.17.0-rc7-00045-g0512e0134582
libvirt:
Compiled against library: libvirt 4.4.0
Using library: libvirt 4.4.0
Using API: QEMU 4.4.0
Running hypervisor: QEMU 2.12.50
#virsh managedsave avocado-vt-vm1
Domain avocado-vt-vm1 state saved by libvirt
# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started
Guest console.
# uname -r
4.17.0-rc7-00045-g0512e0134582
This bug can be closed.
libvirt compiled against 105bcdde76bc8c64f2d9aca9db684186a5e96e63
|