1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
|
graphic: 0.706
device: 0.695
register: 0.680
performance: 0.679
architecture: 0.510
semantic: 0.459
user-level: 0.369
hypervisor: 0.343
mistranslation: 0.340
peripherals: 0.331
ppc: 0.330
permissions: 0.325
x86: 0.281
network: 0.270
VMM: 0.258
i386: 0.233
assembly: 0.232
boot: 0.197
socket: 0.193
kernel: 0.187
debug: 0.162
files: 0.156
virtual: 0.155
vnc: 0.148
risc-v: 0.146
KVM: 0.137
arm: 0.119
PID: 0.110
TCG: 0.065
Race condition during QEMU exit cleanup can lead to deadlock
Description of problem:
During the cleanup phase of QEMU exiting, there is a small race condition window that can lead QEMU to lock up completely:
In the main QEMU thread, during the exit, the thread will execute the 'qemu_cleanup' function, which calls 'do_vm_stop', which calls 'pause_all_vcpus'. This method tries to (as the name suggests) stop/pause all the vcpu threads. At the same time, the vcpu thread might have just existed it's main mttcg exec loop, which means it will enter 'qemu_wait_io_event'. At this point, the following race condition can occur:
- vcpu_thread - cpus.c:416 <= enters qemu_wait_io_event
- shutdown_thread - cpus.c:555 <= enters pause_all_vcpus
- vcpu_thread - cpus.c:418 <= cpu_thread_is_idle returns true, cpu->stop not set yet
- shutdown_thread - cpus.c:560/561 <= sets cpu->stop and kicks the vcpu, but it's not waiting on cpu->halt_cond yet, so nothing happens
- vcpu_thread - cpus.c:423 <= starts waiting on cpu->halt_cond
- shutdown_thread - cpus.c:570 <= not all vcpus paused, so enters while loop
- shutdown_thread - cpus.c:571 <= starts waiting on qemu_pause_cond
- **deadlock**
In my case, my plugin registers qemu_plugin_vcpu_idle_cb, so the race window is extended significantly in the vcpu thread (cpus.c:421) but I believe it can happen with the smaller race window as well.
Note that this explanation is just based on my understanding of the code, and the final state of QEMU during the deadlock after I attached: The main thread (thread 1) was waiting on qemu_pause_cond in pause_all_vcpus, and the vcpu was waiting on cpu->halt_cond in qemu_wait_io_event, with no one else to wake either of them up. (This was following an exit that was triggered by a timeout signal)
Steps to reproduce:
This is a race condition, so I don't have a reliable reproducer.
|