1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
|
id = 2634
title = "Replay/record does not work with `rrsnapshot`/`loadvm`"
state = "opened"
created_at = "2024-10-21T13:18:30.992Z"
closed_at = "n/a"
labels = ["accel: TCG", "icount"]
url = "https://gitlab.com/qemu-project/qemu/-/issues/2634"
host-os = "Ubuntu 20.04.6 LTS"
host-arch = "x86_64"
qemu-version = "9.1.0"
guest-os = "alpine-standard-3.20.3-x86_64.iso"
guest-arch = "x86_64"
description = """Qemu's record/replay feature does not properly work when using snapshots (like rrsnapshot).
Record/replay without snapshotting works just fine, but when using `rrsnapshot=...` the replay is stuck at boot. `loadvm` monitor command also gets qemu stuck.
Record command:
```
$ qemu-system-x86_64 \\
-cpu SandyBridge -smp 1 \\
-serial stdio -display none \\
-m 4096 \\
-drive file=./empty.qcow2,id=rr \\
-kernel ./boot/vmlinuz-lts \\
-initrd ./boot/initramfs-lts .
-monitor telnet::12345,server,nowait \\
-append "console=ttyS0 root=/dev/ram0 alpine_dev=cdrom:iso9660 modules=loop,squashfs,sd-mod,usb-storage quiet" \\
-icount shift=auto,rrfile=rr,rr=record,rrsnapshot=init
```
Broken replay command, which gets qemu stuck:
```
$ qemu-system-x86_64 \\
-cpu SandyBridge -smp 1 \\
-serial stdio -display none \\
-m 4096 \\
-drive file=./empty.qcow2,id=rr \\
-kernel ./boot/vmlinuz-lts \\
-initrd ./boot/initramfs-lts .
-monitor telnet::12345,server,nowait \\
-append "console=ttyS0 root=/dev/ram0 alpine_dev=cdrom:iso9660 modules=loop,squashfs,sd-mod,usb-storage quiet" \\
-icount shift=auto,rrfile=rr,rr=replay,rrsnapshot=init
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
```
Record/replay without `rrsnapshot`/`loadvm`/etc works as expected."""
reproduce = """To reproduce i've used alpine linux kernel as the guest:
```
wget https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-standard-3.20.3-x86_64.iso
7z x alpine-standard-3.20.3-x86_64.iso
```
Prerequisites - an empty qcow2 file for snapshots:
```
qemu-img create -f qcow2 empty.qcow2 1G
```
Running an alpine linux kernel with `rr=record` - works just fine, kernel boots, accepts input.
```
$ qemu-system-x86_64 \\
-cpu SandyBridge -smp 1 \\
-serial stdio -display none \\
-m 4096 \\
-drive file=./empty.qcow2,id=rr \\
-kernel ./boot/vmlinuz-lts \\
-initrd ./boot/initramfs-lts .
-monitor telnet::12345,server,nowait \\
-append "console=ttyS0 root=/dev/ram0 alpine_dev=cdrom:iso9660 modules=loop,squashfs,sd-mod,usb-storage quiet" \\
-icount shift=auto,rrfile=rr,rr=record,rrsnapshot=init
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
mount: mounting /dev/ram0 on /sysroot failed: Invalid argument
Mounting root failed.
initramfs emergency recovery shell launched. Type 'exit' to continue boot
sh: can't access tty; job control turned off
~ # ls -alh
total 32K
drwx------ 18 root root 0 Oct 21 13:02 .
drwx------ 18 root root 0 Oct 21 13:02 ..
-rw------- 1 root root 8 Oct 21 13:02 .ash_history
drwxr-xr-x 2 root root 0 Jun 18 12:44 .modloop
drwxr-xr-x 2 root root 0 Oct 21 13:02 bin
drwxr-xr-x 9 root root 2.5K Oct 21 13:02 dev
drwxr-xr-x 4 root root 0 Oct 21 13:02 etc
-rwxr-xr-x 1 root root 25.9K Jun 18 12:44 init
drwxr-xr-x 5 root root 0 Jun 18 12:44 lib
drwxr-xr-x 5 root root 0 Jun 18 12:44 media
drwxr-xr-x 2 root root 0 Jun 18 12:44 newroot
dr-xr-xr-x 114 root root 0 Oct 21 13:02 proc
drwx------ 2 root root 0 Sep 4 12:53 root
drwxr-xr-x 3 root root 0 Oct 21 13:02 run
drwxr-xr-x 2 root root 0 Oct 21 13:02 sbin
dr-xr-xr-x 13 root root 0 Oct 21 13:02 sys
drwxr-xr-x 2 root root 0 Oct 21 13:02 sysroot
drwxr-xr-x 2 root root 0 Oct 21 13:02 tmp
drwxr-xr-x 5 root root 0 Oct 21 13:02 usr
drwxr-xr-x 3 root root 0 Jun 18 12:44 var
~ # echo "AAAAAAAA?"
AAAAAAAA?
~ #
```
`rr`-file is produced, which can be used for replaying **without** `rrsnapshot`-option:
```
$ qemu-system-x86_64 \\
-cpu SandyBridge -smp 1 \\
-serial stdio -display none \\
-m 4096 \\
-drive file=./empty.qcow2,id=rr \\
-kernel ./boot/vmlinuz-lts \\
-initrd ./boot/initramfs-lts .
-monitor telnet::12345,server,nowait \\
-append "console=ttyS0 root=/dev/ram0 alpine_dev=cdrom:iso9660 modules=loop,squashfs,sd-mod,usb-storage quiet" \\
-icount shift=auto,rrfile=rr,rr=replay
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
mount: mounting /dev/ram0 on /sysroot failed: Invalid argument
Mounting root failed.
initramfs emergency recovery shell launched. Type 'exit' to continue boot
sh: can't access tty; job control turned off
~ # ls -alh
total 32K
drwx------ 18 root root 0 Oct 21 13:02 .
drwx------ 18 root root 0 Oct 21 13:02 ..
-rw------- 1 root root 8 Oct 21 13:02 .ash_history
drwxr-xr-x 2 root root 0 Jun 18 12:44 .modloop
drwxr-xr-x 2 root root 0 Oct 21 13:02 bin
drwxr-xr-x 9 root root 2.5K Oct 21 13:02 dev
drwxr-xr-x 4 root root 0 Oct 21 13:02 etc
-rwxr-xr-x 1 root root 25.9K Jun 18 12:44 init
drwxr-xr-x 5 root root 0 Jun 18 12:44 lib
drwxr-xr-x 5 root root 0 Jun 18 12:44 media
drwxr-xr-x 2 root root 0 Jun 18 12:44 newroot
dr-xr-xr-x 114 root root 0 Oct 21 13:02 proc
drwx------ 2 root root 0 Sep 4 12:53 root
drwxr-xr-x 3 root root 0 Oct 21 13:02 run
drwxr-xr-x 2 root root 0 Oct 21 13:02 sbin
dr-xr-xr-x 13 root root 0 Oct 21 13:02 sys
drwxr-xr-x 2 root root 0 Oct 21 13:02 sysroot
drwxr-xr-x 2 root root 0 Oct 21 13:02 tmp
drwxr-xr-x 5 root root 0 Oct 21 13:02 usr
drwxr-xr-x 3 root root 0 Jun 18 12:44 var
~ # echo "AAAAAAAA?"
AAAAAAAA?
~ #
```
As you can see, replaying emulation session works as expected. How ever, if I add the `rrsnapshot`-option, it gets stuck:
```
$ qemu-system-x86_64 \\
-cpu SandyBridge -smp 1 \\
-serial stdio -display none \\
-m 4096 \\
-drive file=./empty.qcow2,id=rr \\
-kernel ./boot/vmlinuz-lts \\
-initrd ./boot/initramfs-lts .
-monitor telnet::12345,server,nowait \\
-append "console=ttyS0 root=/dev/ram0 alpine_dev=cdrom:iso9660 modules=loop,squashfs,sd-mod,usb-storage quiet" \\
-icount shift=auto,rrfile=rr,rr=replay,rrsnapshot=init
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
```
This also can be reproduced without `rrsnapshot` option, by issuing `loadvm init` from qemu monitor:
```
$ telnet localhost 12345
qemu> loadvm init
...
```
Or, by using `gdb` and issuing reverse-commands that require `loadvm` to load previous state, like `reverse-stepi` or `reverse-continue`.
Attaching a debugger & using debug-prints shows some thread being stuck in the [`rcu.c`](https://gitlab.com/qemu-project/qemu/-/blob/master/util/rcu.c), near the `qemu_event_wait(&rcu_call_ready_event);`. I've tried to wait for quite some time (about an hour) and there was no result."""
additional = """**Qemu build.** Qemu binary built from sources of 9.1.0 with `--target-list=x86_64-softmmu`.
**Host machine.** An almost clean Ubuntu 20.04 with necessary packages for building qemu from the latest release sources."""
|