1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
|
graphic: 0.941
semantic: 0.938
mistranslation: 0.923
other: 0.920
instruction: 0.901
assembly: 0.887
socket: 0.850
device: 0.845
vnc: 0.838
network: 0.744
boot: 0.707
KVM: 0.666
The thread of "CPU 0 /KVM" keeping 99.9%CPU
Hi Expert:
The VM is hung here after (2, or 3, or 5 and the longest time is 10 hours) by qemu-kvm.
Notes:
for VM:
OS: RHEL 7.6
CPU: 1
MEM:4G
For qemu-kvm:
1) version:
/usr/libexec/qemu-kvm -version
QEMU emulator version 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.4.1)
2) once the issue is occurred, the CPU of "CPU0 /KVM" is more than 99% by com "top -p VM_pro_ID"
PID UDER PR NI RES S % CPU %MEM TIME+ COMMAND
872067 qemu 20 0 1.6g R 99.9 0.6 37:08.87 CPU 0/KVM
3) use "pstack 493307" and below is function trace
Thread 1 (Thread 0x7f2572e73040 (LWP 872067)):
#0 0x00007f256cad8fcf in ppoll () from /lib64/libc.so.6
#1 0x000055ff34bdf4a9 in qemu_poll_ns ()
#2 0x000055ff34be02a8 in main_loop_wait ()
#3 0x000055ff348bfb1a in main ()
4) use strace "strace -tt -ff -p 872067 -o cfx" and below log keep printing
21:24:02.977833 ppoll([{fd=4, events=POLLIN}, {fd=6, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLIN}, {fd=80, events=POLLIN}, {fd=82, events=POLLIN}, {fd=84, events=POLLIN}, {fd=115, events=POLLIN}, {fd=121, events=POLLIN}], 9, {0, 0}, NULL, 8) = 0 (Timeout)
21:24:02.977918 ppoll([{fd=4, events=POLLIN}, {fd=6, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLIN}, {fd=80, events=POLLIN}, {fd=82, events=POLLIN}, {fd=84, events=POLLIN}, {fd=115, events=POLLIN}, {fd=121, events=POLLIN}], 9, {0, 911447}, NULL, 8) = 0 (Timeout)
21:24:02.978945 ppoll([{fd=4, events=POLLIN}, {fd=6, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLIN}, {fd=80, events=POLLIN}, {fd=82, events=POLLIN}, {fd=84, events=POLLIN}, {fd=115, events=POLLIN}, {fd=121, events=POLLIN}], 9, {0, 0}, NULL, 8) = 0 (Timeout)
Therefore, I think the thread "CPU 0/KVM" is in tight loop.
5) use reset can recover this issue. however, it will reoccurred again.
Current work around is increase one CPU for this VM, then issue is gone.
thanks
Cliff
one changes:
Guest VM is Red Hat Enterprise Linux 8.1 (Ootpa).
there is no issue once guest VM is RHEL7.6.
Appreciate any comments or clues
Can you try with a newer version of CentOS? I think there should be newer versions of qemu-kvm-ev available, so maybe the problem is gone there.
Otherwise, please either try to reproduce this problem with upstream QEMU, or report it to the CentOS bug tracker (https://bugs.centos.org/), since we do not provide support for distribution builds in the upstream QEMU project here.
Hi Thomas,
Do you have any quick suggestion before report it on CentOS?
thanks
Cliff
I think you should definitely try a newer version if available - otherwise they'll likely refuse to help you, too (nobody wants to debug old versions when bugs are already fixed in newer ones)
Got it!
BTW, you can confirm this is bug for qemu-kvm, right?
thank you!
Cliff
Add the ticket link in centos
https://bugs.centos.org/view.php?id=17385
[Expired for QEMU because there has been no activity for 60 days.]
|