1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
|
id = 1799
title = "Support running real-world Android on Arm by supporting one-register list for the POP (LDMIA) Thumb32 instruction."
state = "closed"
created_at = "2023-07-28T22:37:39.920Z"
closed_at = "2023-10-31T17:39:10.200Z"
labels = ["Bite Sized", "accel: TCG", "target: arm", "workflow::Patch available"]
url = "https://gitlab.com/qemu-project/qemu/-/issues/1799"
host-os = "Linux"
host-arch = "x86_64"
qemu-version = "8.0.3"
guest-os = "LineageOS 18.1 on Waydroid 1.4.1 on PostmarketOS edge"
guest-arch = "aarch64"
description = "n/a"
reproduce = """1. Get any aarch64 Linux on QEMU for x86_64 running. Make sure that Wayland is running. (For example, build PostmarketOS with "phosh" for aarch64 and install it.)
2. Install waydroid (e.g. `apk add waydroid`).
3. Install the LineageOS 18.1 for waydroid image (e.g. `waydroid init`).
4. Run the waydroid-container (e.g. `rc-service waydroid-container restart`).
5. Start the waydroid session (e.g. click on the "Waydroid" symbol on the graphical user interface).
6. Observe the waydroid log file (e.g. run `waydroid logcat`)."""
additional = """The output of the Android log (using `waydroid logcat`) will be akin:
```
23908 23908 D AndroidRuntime: >>>>>> START com.android.internal.os.ZygoteInit uid 0 <<<<<<
23908 23908 I AndroidRuntime: Using default boot image
23908 23908 I AndroidRuntime: Leaving lock profiling enabled
23908 23908 E cutils-trace: Error opening trace file: No such file or directory (2)
23908 23908 I zygote : option[0]=-Xzygote
23908 23908 I zygote : option[1]=exit
23908 23908 I zygote : option[2]=vfprintf
23908 23908 I zygote : option[3]=sensitiveThread
23908 23908 I zygote : option[4]=-verbose:gc
23908 23908 I zygote : option[5]=-XX:PerfettoHprof=true
23908 23908 I zygote : option[6]=-Xms8m
23908 23908 I zygote : option[7]=-Xmx512m
23908 23908 I zygote : option[8]=-XX:HeapGrowthLimit=192m
23908 23908 I zygote : option[9]=-XX:HeapMinFree=8m
23908 23908 I zygote : option[10]=-XX:HeapMaxFree=16m
23908 23908 I zygote : option[11]=-XX:HeapTargetUtilization=0.6
23908 23908 I zygote : option[12]=-Xusejit:true
23908 23908 I zygote : option[13]=-Xjitsaveprofilinginfo
23908 23908 I zygote : option[14]=-XjdwpOptions:suspend=n,server=y
23908 23908 I zygote : option[15]=-XjdwpProvider:default
23908 23908 I zygote : option[16]=-Xopaque-jni-ids:swapable
23908 23908 I zygote : option[17]=-Xlockprofthreshold:500
23908 23908 I zygote : option[18]=-Xcompiler-option
23908 23908 I zygote : option[19]=--instruction-set-variant=generic
23908 23908 I zygote : option[20]=-Xcompiler-option
23908 23908 I zygote : option[21]=--instruction-set-features=default
23908 23908 I zygote : option[22]=-Xcompiler-option
23908 23908 I zygote : option[23]=--generate-mini-debug-info
23908 23908 I zygote : option[24]=-Ximage-compiler-option
23908 23908 I zygote : option[25]=--runtime-arg
23908 23908 I zygote : option[26]=-Ximage-compiler-option
23908 23908 I zygote : option[27]=-Xms64m
23908 23908 I zygote : option[28]=-Ximage-compiler-option
23908 23908 I zygote : option[29]=--runtime-arg
23908 23908 I zygote : option[30]=-Ximage-compiler-option
23908 23908 I zygote : option[31]=-Xmx64m
23908 23908 I zygote : option[32]=-Ximage-compiler-option
23908 23908 I zygote : option[33]=--dirty-image-objects=/system/etc/dirty-image-objects
23908 23908 I zygote : option[34]=-Ximage-compiler-option
23908 23908 I zygote : option[35]=--instruction-set-variant=generic
23908 23908 I zygote : option[36]=-Ximage-compiler-option
23908 23908 I zygote : option[37]=--instruction-set-features=default
23908 23908 I zygote : option[38]=-Ximage-compiler-option
23908 23908 I zygote : option[39]=--generate-mini-debug-info
23908 23908 I zygote : option[40]=-Duser.locale=en-US
23908 23908 I zygote : option[41]=--cpu-abilist=armeabi-v7a,armeabi
23908 23908 I zygote : option[42]=-Xcore-platform-api-policy:just-warn
23908 23908 I zygote : option[43]=-Xfingerprint:waydroid/lineage_waydroid_arm64/waydroid_arm64:11/RQ3A.211001.001/48:userdebug/test-keys
23908 23908 I zygote : Core platform API reporting enabled, enforcing=false
23908 23908 D zygote : Time zone APEX ICU file found: /apex/com.android.tzdata/etc/icu/icu_tzdata.dat
23908 23908 D zygote : I18n APEX ICU file found: /apex/com.android.i18n/etc/icu/icudt66l.dat
23908 23908 I zygote : Using memfd for future sealing
23908 23908 W zygote : Using default instruction set features for ARM CPU variant (generic) using conservative defaults
49 49 I tombstoned: received crash request for pid 23908
23908 23908 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
23908 23908 F DEBUG : LineageOS Version: '18.1-20230723-VANILLA-waydroid_arm64'
23908 23908 F DEBUG : Build fingerprint: 'waydroid/lineage_waydroid_arm64/waydroid_arm64:11/RQ3A.211001.001/48:userdebug/test-keys'
23908 23908 F DEBUG : Revision: '0'
23908 23908 F DEBUG : ABI: 'arm'
23908 23908 F DEBUG : Timestamp: 2023-07-28 14:13:34+0000
23908 23908 F DEBUG : pid: 23908, tid: 23908, name: main >>> zygote <<<
23908 23908 F DEBUG : uid: 0
23908 23908 F DEBUG : signal 4 (SIGILL), code 1 (ILL_ILLOPC), fault addr 0x709443da (*pc=0x4000e8bd)
23908 23908 F DEBUG : r0 54647764 r1 3fb9709b r2 fffffe56 r3 4337ffff
23908 23908 F DEBUG : r4 707184b0 r5 3fdaaaaa r6 f295837e r7 00000001
23908 23908 F DEBUG : r8 00000000 r9 f7986e00 r10 ffa33320 r11 ffa332e4
23908 23908 F DEBUG : ip e9930ba4 sp ffa332cc lr 709443d5 pc 709443da
23908 23908 F DEBUG :
23908 23908 F DEBUG : backtrace:
23908 23908 F DEBUG : #00 pc 0007e3da /apex/com.android.art/javalib/arm/boot.oat (art_jni_trampoline+34) (BuildId: 4af94ec040111dd87be55d34780e36769428675c)
23908 23908 F DEBUG : #01 pc 000d39d5 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #02 pc 004f0759 /apex/com.android.art/lib/libart.so (art_quick_invoke_static_stub+276) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #03 pc 0012ca93 /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+166) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #04 pc 00240bbf /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #05 pc 002388df /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+746) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #06 pc 004e44db /apex/com.android.art/lib/libart.so (MterpInvokeStatic+482) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #07 pc 000ce594 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #08 pc 003bdaa0 /system/framework/framework.jar
23908 23908 F DEBUG : #09 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #10 pc 00238109 /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #11 pc 00239581 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+536) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #12 pc 004e7239 /apex/com.android.art/lib/libart.so (MterpInvokeStaticRange+372) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #13 pc 000ce894 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #14 pc 003bd9d4 /system/framework/framework.jar
23908 23908 F DEBUG : #15 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #16 pc 00238109 /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #17 pc 00239581 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+536) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #18 pc 004e7239 /apex/com.android.art/lib/libart.so (MterpInvokeStaticRange+372) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #19 pc 000ce894 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #20 pc 003bc286 /system/framework/framework.jar
23908 23908 F DEBUG : #21 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #22 pc 00238109 /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #23 pc 002388c7 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+722) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #24 pc 004e44db /apex/com.android.art/lib/libart.so (MterpInvokeStatic+482) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #25 pc 000ce594 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #26 pc 003b1c7c /system/framework/framework.jar
23908 23908 F DEBUG : #27 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #28 pc 0023803d /apex/com.android.art/lib/libart.so (art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*)+120) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #29 pc 004d321b /apex/com.android.art/lib/libart.so (artQuickToInterpreterBridge+686) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #30 pc 000d8561 /apex/com.android.art/lib/libart.so (art_quick_to_interpreter_bridge+32) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #31 pc 0042dbaf /system/framework/arm/boot-framework.oat (android.graphics.ColorSpace$Rgb.isSrgb+446) (BuildId: 7ce3c24f3f20164927036fc8f58e1baa2a8f4020)
23908 23908 F DEBUG : #32 pc 0042cddf /system/framework/arm/boot-framework.oat (android.graphics.ColorSpace$Rgb.<init>+822) (BuildId: 7ce3c24f3f20164927036fc8f58e1baa2a8f4020)
23908 23908 F DEBUG : #33 pc 000d39d5 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #34 pc 004f0627 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub+282) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #35 pc 0012ca81 /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+148) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #36 pc 00240bbf /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #37 pc 00239597 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+558) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #38 pc 004e6b7d /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+392) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #39 pc 000ce814 /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #40 pc 003bce74 /system/framework/framework.jar
23908 23908 F DEBUG : #41 pc 004e6cdd /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+744) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #42 pc 000ce814 /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #43 pc 003bce8c /system/framework/framework.jar
23908 23908 F DEBUG : #44 pc 004e6cdd /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+744) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #45 pc 000ce814 /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #46 pc 003be6b6 /system/framework/framework.jar
23908 23908 F DEBUG : #47 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #48 pc 0023803d /apex/com.android.art/lib/libart.so (art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*)+120) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #49 pc 004d321b /apex/com.android.art/lib/libart.so (artQuickToInterpreterBridge+686) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #50 pc 000d8561 /apex/com.android.art/lib/libart.so (art_quick_to_interpreter_bridge+32) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #51 pc 000d39d5 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #52 pc 004f0759 /apex/com.android.art/lib/libart.so (art_quick_invoke_static_stub+276) (BuildId: d0f40e4862987997ffa9c0a264e61174)
23908 23908 F DEBUG : #53 pc 0012ca93 /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+166) (BuildId: d0f40e4862987997ffa9c0a264e61174)
```
Analyzing with `gdb` (by repeatedly calling `gdb -p "$(ps xua | grep zygote | grep -v grep | grep -v zygote64 | awk {'print $2'})"` until `gdb` attaches earlier to the current `zygote` process than the offending instruction is reached) reveals that the crash happens here:
```
0x6fc373b0 <+944>: cmp r3, #223 @ 0xdf
0x6fc373b2 <+946>: movs r6, r0
0x6fc373b4 <+948>: movs r0, r5
0x6fc373b6 <+950>: movs r0, r0
0x6fc373b8 <+952>: push {lr}
0x6fc373ba <+954>: sub sp, #4
0x6fc373bc <+956>: vstr d0, [sp, #12]
0x6fc373c0 <+960>: vstr d1, [sp, #20]
0x6fc373c4 <+964>: mov r4, r0
0x6fc373c6 <+966>: ldr r2, [sp, #20]
0x6fc373c8 <+968>: ldr r3, [sp, #24]
0x6fc373ca <+970>: ldr r0, [sp, #12]
0x6fc373cc <+972>: ldr r1, [sp, #16]
0x6fc373ce <+974>: ldr.w r12, [r4, #20]
0x6fc373d2 <+978>: blx r12
0x6fc373d4 <+980>: vmov d0, r0, r1
0x6fc373d8 <+984>: add sp, #4
=> 0x6fc373da <+986>: ldmia.w sp!, {lr}
0x6fc373de <+990>: bx lr
```
(note that the actual address changes for every instance of `zygote`, probably due to address-space layout randomization)
The instruction at this location is 0xe8bd4000, as evidenced by:
```
(gdb) x/16hx 0x6fc373da
0x6fc373da <oatexec+986>: 0xe8bd 0x4000 0x4770 0x2c0f 0x0006 0x0020 0x0000 0xb500
0x6fc373ea <oatexec+1002>: 0xb081 0xed8d 0x0b03 0x4604 0x9803 0x9904 0xf8d4 0xc014
```
The disassembly into `ldmia.w sp!, {lr}` is indeed correct. However, such an instruction [would be assembled](https://developer.arm.com/documentation/ddi0308/d/Thumb-Instructions/Alphabetical-list-of-Thumb-instructions/POP?lang=en) into `pop lr` and then into `ldr.w lr,[sp,#-4]`, which would be encoded differently. Hence, the assembly into this instruction was incorrect in the first place.
It turns out that the assembly error is due to an error in the [`vixl` ARMv8 Runtime Code Generation Library](https://github.com/Linaro/vixl), which is also used by Android. This error [has been fixed by Feb 9, 2021](https://github.com/Linaro/vixl/commit/b0a2e281aebbf93e6ee521dcc40ba6dd2aa5124d). However, this fix has [not made it into Android 13](https://android.googlesource.com/platform/external/vixl/+log/02ab12aafeb5278d89184ae6a3ff3a7883b34c5e). Thus, at least Android 11, Android 12, Android 13 cannot run on current `qemu-system-aarch64`, while it should.
Users of the Android emulator (also based on QEMU) do not seem to suffer from this bug because the Android QEMU [has bitrotted since the year 2018](https://android.googlesource.com/platform/external/qemu/+log/e7390f2265257d66093dfe858ce3a47b2e1de539/target/arm/translate.c) and hence has not seen any Arm emulation modernization in QEMU (e.g. the Tiny Code Generator) since, and only this modernization has exposed this bug in the first place."""
|