Support running real-world Android on Arm by supporting one-register list for the POP (LDMIA) Thumb32 instruction. Steps to reproduce: 1. Get any aarch64 Linux on QEMU for x86_64 running. Make sure that Wayland is running. (For example, build PostmarketOS with "phosh" for aarch64 and install it.) 2. Install waydroid (e.g. `apk add waydroid`). 3. Install the LineageOS 18.1 for waydroid image (e.g. `waydroid init`). 4. Run the waydroid-container (e.g. `rc-service waydroid-container restart`). 5. Start the waydroid session (e.g. click on the "Waydroid" symbol on the graphical user interface). 6. Observe the waydroid log file (e.g. run `waydroid logcat`). Additional information: The output of the Android log (using `waydroid logcat`) will be akin: ``` 23908 23908 D AndroidRuntime: >>>>>> START com.android.internal.os.ZygoteInit uid 0 <<<<<< 23908 23908 I AndroidRuntime: Using default boot image 23908 23908 I AndroidRuntime: Leaving lock profiling enabled 23908 23908 E cutils-trace: Error opening trace file: No such file or directory (2) 23908 23908 I zygote : option[0]=-Xzygote 23908 23908 I zygote : option[1]=exit 23908 23908 I zygote : option[2]=vfprintf 23908 23908 I zygote : option[3]=sensitiveThread 23908 23908 I zygote : option[4]=-verbose:gc 23908 23908 I zygote : option[5]=-XX:PerfettoHprof=true 23908 23908 I zygote : option[6]=-Xms8m 23908 23908 I zygote : option[7]=-Xmx512m 23908 23908 I zygote : option[8]=-XX:HeapGrowthLimit=192m 23908 23908 I zygote : option[9]=-XX:HeapMinFree=8m 23908 23908 I zygote : option[10]=-XX:HeapMaxFree=16m 23908 23908 I zygote : option[11]=-XX:HeapTargetUtilization=0.6 23908 23908 I zygote : option[12]=-Xusejit:true 23908 23908 I zygote : option[13]=-Xjitsaveprofilinginfo 23908 23908 I zygote : option[14]=-XjdwpOptions:suspend=n,server=y 23908 23908 I zygote : option[15]=-XjdwpProvider:default 23908 23908 I zygote : option[16]=-Xopaque-jni-ids:swapable 23908 23908 I zygote : option[17]=-Xlockprofthreshold:500 23908 23908 I zygote : option[18]=-Xcompiler-option 23908 23908 I zygote : option[19]=--instruction-set-variant=generic 23908 23908 I zygote : option[20]=-Xcompiler-option 23908 23908 I zygote : option[21]=--instruction-set-features=default 23908 23908 I zygote : option[22]=-Xcompiler-option 23908 23908 I zygote : option[23]=--generate-mini-debug-info 23908 23908 I zygote : option[24]=-Ximage-compiler-option 23908 23908 I zygote : option[25]=--runtime-arg 23908 23908 I zygote : option[26]=-Ximage-compiler-option 23908 23908 I zygote : option[27]=-Xms64m 23908 23908 I zygote : option[28]=-Ximage-compiler-option 23908 23908 I zygote : option[29]=--runtime-arg 23908 23908 I zygote : option[30]=-Ximage-compiler-option 23908 23908 I zygote : option[31]=-Xmx64m 23908 23908 I zygote : option[32]=-Ximage-compiler-option 23908 23908 I zygote : option[33]=--dirty-image-objects=/system/etc/dirty-image-objects 23908 23908 I zygote : option[34]=-Ximage-compiler-option 23908 23908 I zygote : option[35]=--instruction-set-variant=generic 23908 23908 I zygote : option[36]=-Ximage-compiler-option 23908 23908 I zygote : option[37]=--instruction-set-features=default 23908 23908 I zygote : option[38]=-Ximage-compiler-option 23908 23908 I zygote : option[39]=--generate-mini-debug-info 23908 23908 I zygote : option[40]=-Duser.locale=en-US 23908 23908 I zygote : option[41]=--cpu-abilist=armeabi-v7a,armeabi 23908 23908 I zygote : option[42]=-Xcore-platform-api-policy:just-warn 23908 23908 I zygote : option[43]=-Xfingerprint:waydroid/lineage_waydroid_arm64/waydroid_arm64:11/RQ3A.211001.001/48:userdebug/test-keys 23908 23908 I zygote : Core platform API reporting enabled, enforcing=false 23908 23908 D zygote : Time zone APEX ICU file found: /apex/com.android.tzdata/etc/icu/icu_tzdata.dat 23908 23908 D zygote : I18n APEX ICU file found: /apex/com.android.i18n/etc/icu/icudt66l.dat 23908 23908 I zygote : Using memfd for future sealing 23908 23908 W zygote : Using default instruction set features for ARM CPU variant (generic) using conservative defaults 49 49 I tombstoned: received crash request for pid 23908 23908 23908 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** 23908 23908 F DEBUG : LineageOS Version: '18.1-20230723-VANILLA-waydroid_arm64' 23908 23908 F DEBUG : Build fingerprint: 'waydroid/lineage_waydroid_arm64/waydroid_arm64:11/RQ3A.211001.001/48:userdebug/test-keys' 23908 23908 F DEBUG : Revision: '0' 23908 23908 F DEBUG : ABI: 'arm' 23908 23908 F DEBUG : Timestamp: 2023-07-28 14:13:34+0000 23908 23908 F DEBUG : pid: 23908, tid: 23908, name: main >>> zygote <<< 23908 23908 F DEBUG : uid: 0 23908 23908 F DEBUG : signal 4 (SIGILL), code 1 (ILL_ILLOPC), fault addr 0x709443da (*pc=0x4000e8bd) 23908 23908 F DEBUG : r0 54647764 r1 3fb9709b r2 fffffe56 r3 4337ffff 23908 23908 F DEBUG : r4 707184b0 r5 3fdaaaaa r6 f295837e r7 00000001 23908 23908 F DEBUG : r8 00000000 r9 f7986e00 r10 ffa33320 r11 ffa332e4 23908 23908 F DEBUG : ip e9930ba4 sp ffa332cc lr 709443d5 pc 709443da 23908 23908 F DEBUG : 23908 23908 F DEBUG : backtrace: 23908 23908 F DEBUG : #00 pc 0007e3da /apex/com.android.art/javalib/arm/boot.oat (art_jni_trampoline+34) (BuildId: 4af94ec040111dd87be55d34780e36769428675c) 23908 23908 F DEBUG : #01 pc 000d39d5 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #02 pc 004f0759 /apex/com.android.art/lib/libart.so (art_quick_invoke_static_stub+276) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #03 pc 0012ca93 /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+166) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #04 pc 00240bbf /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #05 pc 002388df /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+746) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #06 pc 004e44db /apex/com.android.art/lib/libart.so (MterpInvokeStatic+482) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #07 pc 000ce594 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static+20) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #08 pc 003bdaa0 /system/framework/framework.jar 23908 23908 F DEBUG : #09 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #10 pc 00238109 /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #11 pc 00239581 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+536) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #12 pc 004e7239 /apex/com.android.art/lib/libart.so (MterpInvokeStaticRange+372) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #13 pc 000ce894 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #14 pc 003bd9d4 /system/framework/framework.jar 23908 23908 F DEBUG : #15 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #16 pc 00238109 /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #17 pc 00239581 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+536) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #18 pc 004e7239 /apex/com.android.art/lib/libart.so (MterpInvokeStaticRange+372) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #19 pc 000ce894 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #20 pc 003bc286 /system/framework/framework.jar 23908 23908 F DEBUG : #21 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #22 pc 00238109 /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+144) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #23 pc 002388c7 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+722) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #24 pc 004e44db /apex/com.android.art/lib/libart.so (MterpInvokeStatic+482) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #25 pc 000ce594 /apex/com.android.art/lib/libart.so (mterp_op_invoke_static+20) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #26 pc 003b1c7c /system/framework/framework.jar 23908 23908 F DEBUG : #27 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #28 pc 0023803d /apex/com.android.art/lib/libart.so (art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*)+120) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #29 pc 004d321b /apex/com.android.art/lib/libart.so (artQuickToInterpreterBridge+686) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #30 pc 000d8561 /apex/com.android.art/lib/libart.so (art_quick_to_interpreter_bridge+32) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #31 pc 0042dbaf /system/framework/arm/boot-framework.oat (android.graphics.ColorSpace$Rgb.isSrgb+446) (BuildId: 7ce3c24f3f20164927036fc8f58e1baa2a8f4020) 23908 23908 F DEBUG : #32 pc 0042cddf /system/framework/arm/boot-framework.oat (android.graphics.ColorSpace$Rgb.+822) (BuildId: 7ce3c24f3f20164927036fc8f58e1baa2a8f4020) 23908 23908 F DEBUG : #33 pc 000d39d5 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #34 pc 004f0627 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub+282) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #35 pc 0012ca81 /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+148) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #36 pc 00240bbf /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #37 pc 00239597 /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+558) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #38 pc 004e6b7d /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+392) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #39 pc 000ce814 /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #40 pc 003bce74 /system/framework/framework.jar 23908 23908 F DEBUG : #41 pc 004e6cdd /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+744) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #42 pc 000ce814 /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #43 pc 003bce8c /system/framework/framework.jar 23908 23908 F DEBUG : #44 pc 004e6cdd /apex/com.android.art/lib/libart.so (MterpInvokeDirectRange+744) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #45 pc 000ce814 /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct_range+20) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #46 pc 003be6b6 /system/framework/framework.jar 23908 23908 F DEBUG : #47 pc 0023182b /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.10727712076471079728)+254) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #48 pc 0023803d /apex/com.android.art/lib/libart.so (art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*)+120) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #49 pc 004d321b /apex/com.android.art/lib/libart.so (artQuickToInterpreterBridge+686) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #50 pc 000d8561 /apex/com.android.art/lib/libart.so (art_quick_to_interpreter_bridge+32) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #51 pc 000d39d5 /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #52 pc 004f0759 /apex/com.android.art/lib/libart.so (art_quick_invoke_static_stub+276) (BuildId: d0f40e4862987997ffa9c0a264e61174) 23908 23908 F DEBUG : #53 pc 0012ca93 /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+166) (BuildId: d0f40e4862987997ffa9c0a264e61174) ``` Analyzing with `gdb` (by repeatedly calling `gdb -p "$(ps xua | grep zygote | grep -v grep | grep -v zygote64 | awk {'print $2'})"` until `gdb` attaches earlier to the current `zygote` process than the offending instruction is reached) reveals that the crash happens here: ``` 0x6fc373b0 <+944>: cmp r3, #223 @ 0xdf 0x6fc373b2 <+946>: movs r6, r0 0x6fc373b4 <+948>: movs r0, r5 0x6fc373b6 <+950>: movs r0, r0 0x6fc373b8 <+952>: push {lr} 0x6fc373ba <+954>: sub sp, #4 0x6fc373bc <+956>: vstr d0, [sp, #12] 0x6fc373c0 <+960>: vstr d1, [sp, #20] 0x6fc373c4 <+964>: mov r4, r0 0x6fc373c6 <+966>: ldr r2, [sp, #20] 0x6fc373c8 <+968>: ldr r3, [sp, #24] 0x6fc373ca <+970>: ldr r0, [sp, #12] 0x6fc373cc <+972>: ldr r1, [sp, #16] 0x6fc373ce <+974>: ldr.w r12, [r4, #20] 0x6fc373d2 <+978>: blx r12 0x6fc373d4 <+980>: vmov d0, r0, r1 0x6fc373d8 <+984>: add sp, #4 => 0x6fc373da <+986>: ldmia.w sp!, {lr} 0x6fc373de <+990>: bx lr ``` (note that the actual address changes for every instance of `zygote`, probably due to address-space layout randomization) The instruction at this location is 0xe8bd4000, as evidenced by: ``` (gdb) x/16hx 0x6fc373da 0x6fc373da : 0xe8bd 0x4000 0x4770 0x2c0f 0x0006 0x0020 0x0000 0xb500 0x6fc373ea : 0xb081 0xed8d 0x0b03 0x4604 0x9803 0x9904 0xf8d4 0xc014 ``` The disassembly into `ldmia.w sp!, {lr}` is indeed correct. However, such an instruction [would be assembled](https://developer.arm.com/documentation/ddi0308/d/Thumb-Instructions/Alphabetical-list-of-Thumb-instructions/POP?lang=en) into `pop lr` and then into `ldr.w lr,[sp,#-4]`, which would be encoded differently. Hence, the assembly into this instruction was incorrect in the first place. It turns out that the assembly error is due to an error in the [`vixl` ARMv8 Runtime Code Generation Library](https://github.com/Linaro/vixl), which is also used by Android. This error [has been fixed by Feb 9, 2021](https://github.com/Linaro/vixl/commit/b0a2e281aebbf93e6ee521dcc40ba6dd2aa5124d). However, this fix has [not made it into Android 13](https://android.googlesource.com/platform/external/vixl/+log/02ab12aafeb5278d89184ae6a3ff3a7883b34c5e). Thus, at least Android 11, Android 12, Android 13 cannot run on current `qemu-system-aarch64`, while it should. Users of the Android emulator (also based on QEMU) do not seem to suffer from this bug because the Android QEMU [has bitrotted since the year 2018](https://android.googlesource.com/platform/external/qemu/+log/e7390f2265257d66093dfe858ce3a47b2e1de539/target/arm/translate.c) and hence has not seen any Arm emulation modernization in QEMU (e.g. the Tiny Code Generator) since, and only this modernization has exposed this bug in the first place.