runtime: 0.446 instruction: 0.357 syscall: 0.197 Go garbage collector crashes when using qemu-x86_64 on an aarch64 host Description of problem: Apps compiled for Go and the Go compiler/tool itself crash when they are run with `qemu-x86_64` on an AARCH64 host system. This was not a problem on QEMU 8.2.x (I bisected, see further down). I also seem to recall that Go 1.21 is fine on QEMU 9.x, so maybe some recent change in Go 1.22 + recent changes in QEMU broke something? The crash from Go seems to be in the garbage collector, I cannot reproduce the issue when I disable the GC with `GOGC=off`. Output from Go when it crashes: ``` $ sudo chroot . go build main.go runtime: lfstack.push invalid packing: node=0xffff6542b2c0 cnt=0x1 packed=0xffff6542b2c00001 -> node=0xffffffff6542b2c0 fatal error: lfstack.push runtime stack: runtime.throw({0xa95b29?, 0x797b1e2a383c?}) runtime/panic.go:1023 +0x5c fp=0xc000515f08 sp=0xc000515ed8 pc=0x43c27c runtime.(*lfstack).push(0x0?, 0xc0005041c0?) runtime/lfstack.go:29 +0x125 fp=0xc000515f48 sp=0xc000515f08 pc=0x40fd45 runtime.(*spanSetBlockAlloc).free(...) runtime/mspanset.go:322 runtime.(*spanSet).reset(0xf46980) runtime/mspanset.go:264 +0x79 fp=0xc000515f78 sp=0xc000515f48 pc=0x437219 runtime.finishsweep_m() runtime/mgcsweep.go:258 +0x8d fp=0xc000515fb8 sp=0xc000515f78 pc=0x42a6cd runtime.gcStart.func2() runtime/mgc.go:685 +0xf fp=0xc000515fc8 sp=0xc000515fb8 pc=0x46e40f runtime.systemstack(0x0) runtime/asm_amd64.s:509 +0x4a fp=0xc000515fd8 sp=0xc000515fc8 pc=0x47442a ```` Steps to reproduce: 0. Use an aarch64 host system! 1. Set up binfmt to use qemu-x86_64: ``` $ cat /proc/sys/fs/binfmt_misc/qemu-x86_64 enabled interpreter /usr/bin/qemu-x86_64 flags: OCF offset 0 magic 7f454c4602010100000000000000000002003e00 mask fffffffffffefe00fffffffffffffffffeffffff ``` 2. Download/extract x86_64 rootfs: ``` $ curl -O https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-minirootfs-3.20.2-x86_64.tar.gz ``` 3. Create example app in the x86_64 rootfs: ``` package main func main() { } ``` 4. Build using chroot: ``` $ sudo chroot /path/to/x86_64/rootfs apk add go $ sudo chroot /path/to/x86_64/rootfs go build main.go runtime: lfstack.push invalid packing: node=0xffff6542b2c0 cnt=0x1 packed=0xffff6542b2c00001 -> node=0xffffffff6542b2c0 fatal error: lfstack.push ... ``` 5. As noted previously, if the Go garbage collector is disabled, then it works, presumably because it avoids the bug(?) in QEMU: ``` $ sudo chroot . env GOGC=off go build main.go # might have to mount /dev to build successfully, but Go doesn't panic! ``` Additional information: I've bisected this exact crash/failure to: ``` commit 2952b642a555207748dd961fcbfdc48f198eebb6 Author: Richard Henderson Date: Tue Feb 13 10:20:27 2024 -1000 linux-user: Split out do_munmap Reviewed-by: Philippe Mathieu-Daudé Signed-off-by: Richard Henderson ``` Though a different crash starts happening at the commit before that one: ``` commit ad87d26e6bb13257409f412224c862fc54025e8b Author: Richard Henderson Date: Tue Jan 2 12:57:55 2024 +1100 linux-user: Do early mmap placement only for reserved_va For reserved_va, place all non-fixed maps then proceed as for MAP_FIXED. Signed-off-by: Richard Henderson ``` FYI @rth7680