qemu-system-ppc64le fails with kvm acceleration (Suspected glibc issue!) qemu-system-ppc64(le) fails when invoked with kvm acceleration with error "illegal instruction" > qemu-system-ppc64(le) -M pseries,accel=kvm Illegal instruction (core dumped) In dmesg: Facility 'SCV' unavailable (12), exception at 0x7624f8134c0c, MSR=900000000280f033 Version-Release number of selected component (if applicable): qemu 5.2.0 Linux kernel 5.11 glibc 2.33 all latest updates as of submitting the bug report How reproducible: Always Steps to Reproduce: 1. Run qemu with kvm acceleration Actual results: Illegal instruction Expected results: Normal VM execution Additional info: The machine is a Raptor Talos II Lite with a Sforza V1 8-core, but was also observed on a Raptor Blackbird with the same processor. This was also observed on Fedora 34 beta, which uses glibc 2.33 Also tested on ArchPOWER (unofficial port of Arch Linux for ppc64le) with glibc 2.33 Fedora 33 and Ubuntu 20.10, both using glibc 2.32 do not have this issue, and downgrading the Linux kernel from 5.11 to 5.4 LTS on ArchPOWER solved the problem. Kernel 5.9 and 5.10 have the same issue when combined with glibc2.33 ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: qemu-system 1:5.2+dfsg-6ubuntu2 ProcVersionSignature: Ubuntu 5.11.0-11.12-generic 5.11.0 Uname: Linux 5.11.0-11-generic ppc64le .sys.firmware.opal.msglog: Error: [Errno 13] Permission denied: '/sys/firmware/opal/msglog' ApportVersion: 2.20.11-0ubuntu60 Architecture: ppc64el CasperMD5CheckResult: pass CurrentDesktop: Unity:Unity7:ubuntu Date: Mon Mar 22 14:48:39 2021 InstallationDate: Installed on 2021-03-22 (0 days ago) InstallationMedia: Ubuntu-Server 21.04 "Hirsute Hippo" - Alpha ppc64el (20210321) KvmCmdLine: COMMAND STAT EUID RUID PID PPID %CPU COMMAND ProcKernelCmdLine: root=UUID=f3d03315-0944-4a02-9c87-09c00eba9fa1 ro ProcLoadAvg: 1.20 0.73 0.46 1/1054 6071 ProcSwaps: Filename Type Size Used Priority /swap.img file 8388544 0 -2 ProcVersion: Linux version 5.11.0-11-generic (buildd@bos02-ppc64el-002) (gcc (Ubuntu 10.2.1-20ubuntu1) 10.2.1 20210220, GNU ld (GNU Binutils for Ubuntu) 2.36.1) #12-Ubuntu SMP Mon Mar 1 19:26:20 UTC 2021 SourcePackage: qemu UpgradeStatus: No upgrade log present (probably fresh install) VarLogDump_list: total 0 acpidump: cpu_cores: Number of cores present = 8 cpu_coreson: Number of cores online = 8 cpu_smt: SMT=4 Status changed to 'Confirmed' because the bug affects multiple users. Since this seems to be broken on all Distributions as soon as the triggering combination of kernel/glibc is present I think we'd want to open that up to upstream qemu for a wider discussion and to also hit the ppc64 architecture experts. Furthermore I'm not entirely sure if this needs to be fixed in qemu, it might instead be the case that instead a fix is needed in glibc. Therefore I'm adding a qemu (upstream) bug task for now to have the bug reported there as well (might be worth for awareness anyway) - but chances are that after some debugging it will turn out to become a glibc issue instead. If only I could break this test out of kvm ioctl into something simpler, then we could then properly file against glibc .... Hi Sadoon, thanks for the report! There isn't much to find about this issue yet. One automatic syscaller crash report [1]. On the emulation side there is [2][3]. On the glibc side we have [4][5] adding the use of it with [6] being a fix. All those seem to be in glibc 2.33 - so I'd expect with [6] it should only be issued on power9 which in turn should HW-support the instruction. I was trying to recreate this on power8 and power9 machines. As expected on power8 just nothing happens (the instruction isn't used due to [6]). TBH I first wondered if these Sforza chips [7][8][9] you mentioned are fully identical to a classic IBM p9 box - but I was indeed able to reproduce the issue just fine on an IBM-sold P9 dmesg: [ 1516.438442] Facility 'SCV' unavailable (12), exception at 0x76c9f84c49a0, MSR=900000000280f033 [ 1516.438472] qemu-system-ppc[42884]: illegal instruction (4) at 76c9f84c49a0 nip 76c9f84c49a0 lr 1f12839d9f0 code 1 in libc-2.33.so[76c9f8380000+220000] [ 1516.438489] qemu-system-ppc[42884]: code: e8010010 7c0803a6 4e800020 60420000 7ca42b78 4bffed65 60000000 38210020 [ 1516.438493] qemu-system-ppc[42884]: code: e8010010 7c0803a6 4e800020 60420000 <44000001> 4bffffb8 60000000 60420000 The chip I used for this test is: Model: 2.2 (pvr 004e 1202) Model name: POWER9, altivec supported The syscall this crashes in belongs to the ioctl (gdb) bt #0 __GI___ioctl (fd=, request=536915584) at ../sysdeps/unix/sysv/linux/powerpc/ioctl.c:56 #1 0x00000cb63ef7d9f0 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7d0f48010010, type=type@entry=536915584) at ../../accel/kvm/kvm-all.c:2654 #2 0x00000cb63ef7dbdc in kvm_cpu_exec (cpu=0x7d0f48010010) at ../../accel/kvm/kvm-all.c:2491 #3 0x00000cb63ee78344 in kvm_vcpu_thread_fn (arg=0x7d0f48010010) at ../../accel/kvm/kvm-cpus.c:49 #4 0x00000cb63f1d14bc in qemu_thread_start (args=) at ../../util/qemu-thread-posix.c:521 #5 0x00007d0f4ac69114 in start_thread (arg=0x7d0f23dfe720) at pthread_create.c:473 #6 0x00007d0f4ab755c0 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:103 And jumping into the code of the __GI___ioctl we can clearly see the scv instruction is indeed there in the executed code path: 0x7ffff66c4984 <__GI___ioctl+292> bl 0x7ffff66c36e8 <__GI___tcgetattr+8> 0x7ffff66c4988 <__GI___ioctl+296> nop 0x7ffff66c498c <__GI___ioctl+300> addi r1,r1,32 0x7ffff66c4990 <__GI___ioctl+304> ld r0,16(r1) 0x7ffff66c4994 <__GI___ioctl+308> mtlr r0 0x7ffff66c4998 <__GI___ioctl+312> blr 0x7ffff66c499c <__GI___ioctl+316> ori r2,r2,0 >0x7ffff66c49a0 <__GI___ioctl+320> scv 0 [1]: https://webcache.googleusercontent.com/search?q=cache:uS0jhPekyqMJ:https://syzkaller-ppc64.appspot.com/text%3Ftag%3DCrashReport%26x%3D17d99883000000+&cd=2&hl=de&ct=clnk&gl=uk [2]: https://git.qemu.org/?p=qemu.git;a=commit;h=3c89b8d6ac5b8728cd7620f9885bd953edd18a11 [3]: https://lists.gnu.org/archive/html/qemu-devel/2021-03/msg05425.html [4]: https://sourceware.org/git/?p=glibc.git;a=commit;h=68ab82f56690ada86ac1e0c46bad06ba189a10ef [5]: https://sourceware.org/git/?p=glibc.git;a=commit;h=41f013cef24884604c303435dd1915be2ea5c0e0 [6]: https://sourceware.org/git/?p=glibc.git;a=commit;h=527c89cd32f8522859f58343be3d3dc8f754b783 [7]: https://wiki.raptorcs.com/wiki/Sforza [8]: https://wiki.raptorcs.com/wiki/Talos_II [9]: https://wiki.raptorcs.com/wiki/POWER9 [10]: https://lwn.net/Articles/822867/ qemu calls this ioctl on ppc64 as: sysdeps/unix/sysv/linux/powerpc/ioctl.c result = INLINE_SYSCALL (ioctl, 3, fd, request, arg); The mapping of macros in sysdeps/unix/sysv/linux/powerpc/sysdep.h seems to be: INTERNAL_SYSCALL -> INTERNAL_SYSCALL_NCS -> TRY_SYSCALL_SCV -> SYSCALL_SCV 76 #define SYSCALL_SCV(nr) \ 77 ({ \ 78 __asm__ __volatile__ \ 79 (".machine \"push\"\n\t" \ 80 ".machine \"power9\"\n\t" \ 81 "scv 0\n\t" \ 82 ".machine \"pop\"\n\t" \ 83 "0:" \ 84 : "=&r" (r0), \ 85 "=&r" (r3), "=&r" (r4), "=&r" (r5), \ 86 "=&r" (r6), "=&r" (r7), "=&r" (r8) \ 87 : ASM_INPUT_##nr \ 88 : "r9", "r10", "r11", "r12", \ 89 "lr", "ctr", "memory"); \ 90 r3; \ 91 }) [10] outlined to use PPC_FEATURE2_SCV but [4] does just that. In addition [6] added power9 machine settings as only on this ISA it is available - like: + .machine "push" + .machine "power9" scv 0 + .machine "pop" Maybe there is some generated "scv 0" left that needs the same [6] treatment? OTOH In a normal test program I can run "scv 0" just fine. But not other scv levels (expected). # cat test.c #include int main() { printf("Hello scv 0\n"); __asm__( "scv 0\n\t" ); printf("survived\n"); __asm__( "scv 1\n\t" ); printf("survived level 1\n"); return 0; } # gcc -Wall -o test test.c ./test Hello scv 0 survived Illegal instruction (core dumped) IIRC .machine is only a psedo-op for the assembler. So it is correct that I can't see it in the live disassembly of gdb. The failing "svc 0" from glibcs __GI___ioctl is 0x00007ffff66c49a0 <+320>: 01 00 00 44 scv 0 And in my test program it is 0x0000000100000848 <+44>: 01 00 00 44 scv 0 Hmm, this is the same opcode but fails in just one of the cases. This might need someone being more an ppc64/glibc expert than me :-/ @Frank - could you modify this bug to become mirrored to IBM for their arch-expertise please? As my other repro-code didn't trigger the issue I looked at qemu again and found that before the failing ioctl->scv call there are plenty other even some very similar (same?) calls that work just fine. I wonder if on guest setup qemu (or e.g. the rom we load) might set some arch-bits or such which then breaks the next "scv 0" call. I attached the full ioctl log here. I might be spoiled by the s390x-POP style to define instructions, but in the following doc about the PowerISA unfortunately there is no list of reasons-for-SIGILL. Therefore I'm out of options on this waiting for someone - most likely IBM - to chime in. https://wiki.raptorcs.com/w/images/f/f5/PowerISA_public.v3.1.pdf You need a kernel with a the following fix for POWER9: commit 25edcc50d76c834479d11fcc7de46f3da4d95121 Author: Fabiano Rosas