Openjdk11+ fails to install on s390x While installing openjdk11 or higher from repo, it crashes while configuring ca-certificates-java. Although `java -version` passes, `jar -version` crashes. Detailed logs attached to this issue. ``` # A fatal error has been detected by the Java Runtime Environment: # # SIGILL (0x4) at pc=0x00000040126f9980, pid=8425, tid=8430 # # JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04) # Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, tiered, compressed oops, g1 gc, linux-s390x) # Problematic frame: # J 4 c1 java.lang.StringLatin1.hashCode([B)I java.base@11.0.10 (42 bytes) @ 0x00000040126f9980 [0x00000040126f9980+0x0000000000000000] # # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.8425) # # An error report file with more information is saved as: # //hs_err_pid8425.log sed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /root/core.10740) # # An error report file with more information is saved as: # /root/hs_err_pid10740.log ``` Observed this on s390x/ubuntu as well as alpine when run on amd64. Please note, on native s390x, the installation is successful. Also this crash is not observed while installing openjdk-8-jdk. Qemu version: 5.2.0 Please let me know if any more details are needed. You don't say how you're invoking QEMU (system emulation? usermode? what command line?) Please give the full commandline, repro steps, and any files/images we would need to reproduce the failure. Please find below steps to reproduce the issue(Running on amd64 VM): ``` apt-get install -y qemu qemu-user-static docker run --rm --privileged multiarch/qemu-user-static --reset -p yes docker run -it s390x/ubuntu:20.04 bash --> apt-get update && apt-get install -y openjdk-11-jdk jar --version ``` Same BUG as https://bugs.launchpad.net/qemu/+bug/1862874 Tried building jdk 11 from source, the generated executable still crashes(fastdebug as well as release mode): ``` root@24d396a17e00:~/jdk# build/linux-s390x-normal-server-release/jdk/bin/java -version # # A fatal error has been detected by the Java Runtime Environment: # # SIGILL (0x4) at pc=0x000000400b234440, pid=18175, tid=18178 # # JRE version: OpenJDK Runtime Environment (11.0) (build 11-internal+0-adhoc..jdk) # Java VM: OpenJDK 64-Bit Server VM (11-internal+0-adhoc..jdk, mixed mode, tiered, compressed oops, g1 gc, linux-s390x) # Problematic frame: # J 78 c1 java.util.HashMap.afterNodeInsertion(Z)V java.base (1 bytes) @ 0x000000400b234440 [0x000000400b234400+0x0000000000000040] # # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /root/jdk/core.18175) # # An error report file with more information is saved as: # /root/jdk/hs_err_pid18175.log Compiled method (c1) 1795 78 3 java.util.HashMap::afterNodeInsertion (1 bytes) total in heap [0x000000400b234210,0x000000400b2345b0] = 928 relocation [0x000000400b234378,0x000000400b2343a0] = 40 constants [0x000000400b2343c0,0x000000400b234400] = 64 main code [0x000000400b234400,0x000000400b234500] = 256 stub code [0x000000400b234500,0x000000400b234558] = 88 metadata [0x000000400b234558,0x000000400b234568] = 16 scopes data [0x000000400b234568,0x000000400b234578] = 16 scopes pcs [0x000000400b234578,0x000000400b2345a8] = 48 dependencies [0x000000400b2345a8,0x000000400b2345b0] = 8 Compiled method (c1) 1806 74 3 java.util.HashMap::putVal (300 bytes) total in heap [0x000000400b230210,0x000000400b231f20] = 7440 relocation [0x000000400b230378,0x000000400b230690] = 792 constants [0x000000400b2306c0,0x000000400b230a00] = 832 main code [0x000000400b230a00,0x000000400b231980] = 3968 stub code [0x000000400b231980,0x000000400b231a68] = 232 metadata [0x000000400b231a68,0x000000400b231ad0] = 104 scopes data [0x000000400b231ad0,0x000000400b231ce8] = 536 scopes pcs [0x000000400b231ce8,0x000000400b231eb8] = 464 dependencies [0x000000400b231eb8,0x000000400b231ec0] = 8 nul chk table [0x000000400b231ec0,0x000000400b231f20] = 96 Could not load hsdis-s390x.so; library not loadable; PrintAssembly is disabled # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Aborted (core dumped) root@24d396a17e00:~/jdk# ``` @davidhildenbrand The other issue which you have mentioned as duplicate shows java getting stuck for long, whereas for me it crashes right away. Do you think these 2 are related? Also observed another behaviour : java -version randomly passes, sometimes. I can also confirm that it is observed under s390x chroot as well(logs below): ``` root@XX:/# ulimit -c unlimited root@XX:/# java -version openjdk version "11.0.10" 2021-01-19 OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.20.04) OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode) root@XX:/# java -version # # A fatal error has been detected by the Java Runtime Environment: # # SIGILL (0x4) at pc=0x0000004012705b40, pid=156601, tid=156604 # # JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04) # Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, tiered, compressed oops, g1 gc, linux-s390x) # Problematic frame: # J 5 c1 java.lang.Object.()V java.base@11.0.10 (1 bytes) @ 0x0000004012705b40 [0x0000004012705b00+0x0000000000000040] # # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.156601) # # An error report file with more information is saved as: # //hs_err_pid156601.log Compiled method (c1) 956 5 3 java.lang.Object:: (1 bytes) total in heap [0x0000004012705910,0x0000004012705cb8] = 936 relocation [0x0000004012705a70,0x0000004012705aa0] = 48 constants [0x0000004012705ac0,0x0000004012705b00] = 64 main code [0x0000004012705b00,0x0000004012705c00] = 256 stub code [0x0000004012705c00,0x0000004012705c58] = 88 metadata [0x0000004012705c58,0x0000004012705c70] = 24 scopes data [0x0000004012705c70,0x0000004012705c80] = 16 scopes pcs [0x0000004012705c80,0x0000004012705cb0] = 48 dependencies [0x0000004012705cb0,0x0000004012705cb8] = 8 Compiled method (c1) 960 5 3 java.lang.Object:: (1 bytes) total in heap [0x0000004012705910,0x0000004012705cb8] = 936 relocation [0x0000004012705a70,0x0000004012705aa0] = 48 constants [0x0000004012705ac0,0x0000004012705b00] = 64 main code [0x0000004012705b00,0x0000004012705c00] = 256 stub code [0x0000004012705c00,0x0000004012705c58] = 88 metadata [0x0000004012705c58,0x0000004012705c70] = 24 scopes data [0x0000004012705c70,0x0000004012705c80] = 16 scopes pcs [0x0000004012705c80,0x0000004012705cb0] = 48 dependencies [0x0000004012705cb0,0x0000004012705cb8] = 8 Could not load hsdis-s390x.so; library not loadable; PrintAssembly is disabled # # If you would like to submit a bug report, please visit: # https://bugs.launchpad.net/ubuntu/+source/openjdk-lts # Aborted (core dumped) root@XX:/# ulimit -c unlimited root@XX:/# java -version # # A fatal error has been detected by the Java Runtime Environment: # # SIGILL (0x4) at pc=0x0000004012706a40, pid=156619, tid=156622 # # JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04) # Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, tiered, compressed oops, g1 gc, linux-s390x) # Problematic frame: # J 4 c1 java.lang.Object.()V java.base@11.0.10 (1 bytes) @ 0x0000004012706a40 [0x0000004012706a00+0x0000000000000040] # . (truncating logs) Aborted (core dumped) root@XX:/# ``` Increasing core limit worked once, but it fails eventually. Could you please share your thoughts and provide some pointers on debugging further? As java -version passes few times, further also checked behaviour of Maven. Observed that mvn -v crashes in a similar fashion, however after setting below: export MAVEN_OPTS="-XX:-TieredCompilation -XX:+UseG1GC -Dcount=1000000" mvn -v always passes. root@XX:/# mvn -v OpenJDK 64-Bit Server VM warning: You have loaded library /apache-maven-3.6.3/lib/jansi-native/linux64/libjansi.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'. Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) Maven home: /apache-maven-3.6.3 Java version: 11.0.7, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-s390x Default locale: en_US, platform encoding: ANSI_X3.4-1968 OS name: "linux", version: "5.4.0-70-generic", arch: "s390x", family: "unix" However what I am really interested in, is mvn clean install command which never passes with above settings. @davidhildenbrand, any help would be appreciated. Hi @davidhildenbrand, I'm on the same team as @nam121 and I've been looking at this issue as well. I think this is the same issue as: https://github.com/multiarch/qemu-user-static/issues/129 I've been running an s390x docker image on a master build (with latest s390x commit from Apr 23) of user mode qemu-s390x-static with some debug logging on: $ sudo docker run -e QEMU_CPU="qemu" -e QEMU_LOG="unimp,guest_errors" -e QEMU_LOG_FILENAME="/s390x/qemu_s390x.log" I ran a simple java program with: $ java -Xcomp -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:PrintAssemblyOptions=hsdis-print-bytes -XX:+LogCompilation -XX:LogFile=java_compilation_log.log Main > java_out.txt and the qemu log contained just one line: unimplemented opcode 0x0000 Note that if the JIT is turned off with 'java -Xint', then all programs I've tried run without problem. The hs_err file reports a SIGILL in the same spot as in the other comments: --- SNIP # A fatal error has been detected by the Java Runtime Environment: # # SIGILL (0x4) at pc=0x00000040126d7680, pid=208, tid=211 # # JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04) # Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, compiled mode, tiered, compressed oops, g1 gc, linux-s390x) # Problematic frame: # J 9 c1 java.lang.String.hashCode()I java.base (49 bytes) @ 0x00000040126d7680 [0x00000040126d7640+0x0000000000000040] --- SNIP --- SNIP Instructions: (pc=0x00000040126d7680) 0x00000040126d7580: 00000040 5f5f4140 00000040 5f5f4140 0x00000040126d7590: 00000040 5f5f4140 00000040 5f5f4140 0x00000040126d75a0: 00000040 5f5f4358 00000040 5f5f4358 0x00000040126d75b0: 00000040 5f5f4358 00000040 5f5f4358 0x00000040126d75c0: 00000040 5f5f4140 00000040 5f5f4140 0x00000040126d75d0: 00000000 00000000 ffffffff ffffffff 0x00000040126d75e0: 00000040 5f5f4140 00000000 00000000 0x00000040126d75f0: ffffffff ffffffff 00000040 5f3fb9d0 0x00000040126d7600: 00000040 12238c00 00000040 12232800 0x00000040126d7610: 00000040 5f3fef18 00000040 12238c00 0x00000040126d7620: 00000040 12235000 00000000 00000000 0x00000040126d7630: 00000000 00000000 00000000 00000000 0x00000040126d7640: b9040009 cc08ffff fff85500 2008a784 # <-- String.hashCode() entry point at 0x00000040126d7640 0x00000040126d7650: 0019a51d 0040c019 12167a80 07f10700 0x00000040126d7660: 07000700 07000700 07000700 07000700 0x00000040126d7670: 07000700 07000700 07000700 07000700 0x00000040126d7680: 0000f000 ec51e3e0 f0080024 b904000f # <-- note 0x0000 at 0x00000040126d7680 0x00000040126d7690: a7fbffa0 e300f000 0024c438 ffffff73 --- SNIP The assembly printed by java looks like: --- SNIP [Entry Point] # {method} {0x000000405f3fb9d0} 'hashCode' '()I' in 'java/lang/String' # [sp+0x60] (sp of caller) 0x00000040126d7640: lgr %r0,%r9 ;...b9040009 ; {no_reloc} 0x00000040126d7644: aih %r0,-8 ;...cc08ffff fff8 0x00000040126d764a: cl %r0,8(%r2) ;...55002008 0x00000040126d764e: je 0x00000040126d7680 ;...a7840019 0x00000040126d7652: llihl %r1,64 ;...a51d0040 0x00000040126d7656: iilf %r1,303463040 ;...c0191216 7a80 0x00000040126d765c: br %r1 ;...07f1 0x00000040126d765e: nopr ;...0700 0x00000040126d7660: nopr ;...0700 0x00000040126d7662: nopr ;...0700 0x00000040126d7664: nopr ;...0700 0x00000040126d7666: nopr ;...0700 0x00000040126d7668: nopr ;...0700 0x00000040126d766a: nopr ;...0700 0x00000040126d766c: nopr ;...0700 0x00000040126d766e: nopr ;...0700 0x00000040126d7670: nopr ;...0700 0x00000040126d7672: nopr ;...0700 0x00000040126d7674: nopr ;...0700 0x00000040126d7676: nopr ;...0700 0x00000040126d7678: nopr ;...0700 0x00000040126d767a: nopr ;...0700 0x00000040126d767c: nopr ;...0700 0x00000040126d767e: nopr ;...0700 [Verified Entry Point] 0x00000040126d7680: tmy -81920(%r15),222 ;...ebdef000 ec51 0x00000040126d7686: stg %r14,8(%r15) ;...e3e0f008 0024 0x00000040126d768c: lgr %r0,%r15 ;...b904000f 0x00000040126d7690: aghi %r15,-96 ;...a7fbffa0 0x00000040126d7694: stg %r0,0(%r15) ;...e300f000 0024 0x00000040126d769a: lgrl %r3,0x00000040126d7580 ;...c438ffff ff73 ; {metadata(method data for {method} {0x000000405f3fb9d0} 'hashCode' '()I' in 'java/lang/String')} --- SNIP so IIUC java says its generating 0xebde at 0x00000040126d7680 instead of 0x0000. Hope the above makes sense. I'm not sure where to go from here so any suggestions would be a great help. From looking at the in_asm logs, it looks like that instruction starting with 0xebde is executed once with no problem but the second time its changed to 0x0000. ... # First Time ---------------- IN: 0x40126d6880:  ebde f000 ec51  tmy      -0x14000(%r15), 0xde 0x40126d6886:  e3e0 f008 0024  stg      %r14, 8(%r15) 0x40126d688c:  b904 000f       lgr      %r0, %r15 0x40126d6890:  a7fb ffa0       aghi     %r15, -0x60 0x40126d6894:  e300 f000 0024  stg      %r0, 0(%r15) 0x40126d689a:  c438 ffff ff73  lgrl     %r3, 0x40126d6780 0x40126d68a0:  5840 30dc       l        %r4, 0xdc(%r3) 0x40126d68a4:  c248 0000 0008  agfi     %r4, 8 0x40126d68aa:  5040 30dc       st       %r4, 0xdc(%r3) 0x40126d68ae:  c0f4 0000 00d1  jg       0x40126d6a50 PSW=mask 0000000180000000 addr 00000040126d6880 cc  CC_OP_LTGT0_64 R00=0000000000000000 R01=00000040126d6880 R02=00000006296f5d20 R03=00000006296f5d20 R04=000000405f45fcd8 R05=00000006000000e8 R06=0000004012169380 R07=0000004002c410e8 R08=0000004004019000 R09=000000405f2d29d0 R10=0000004002c41048 R11=00000006296095e0 R12=000000400280ec50 R13=0000004002c411d0 R14=00000040126d5c64 R15=0000004002c40e88 ... # Second Time unimplemented opcode 0x0000 ---------------- IN: PSW=mask 0000000180000000 addr 00000040126d6880 cc CC_OP_LTUGTU_32 R00=0000000000001808 R01=00000040126d53c0 R02=00000006296f5d78 R03=00000006296f5d78 R04=000000405f45fcd8 R05=00000006000000f0 R06=0000004012114000 R07=5f9dbb3700003030 R08=0000004004019000 R09=0000000800001808 R10=0000004002c41048 R11=00000006296095e0 R12=000000400280ec50 R13=0000004002c411d0 R14=00000040126d5c64 R15=0000004002c40e88 Some more analysis: Tried to explicitely compile as well as exclude few methods during compilation such as 'java.lang.StringLatin1::hashCode', 'java.util.concurrent.ConcurrentHashMap', 'java.lang.String*' which are part of trace as logged in above comments, with the help of advanced JIT options. However it is not good enough to draw any conclusion as `java -version` command passes on random runs. `mvn -v` which consistently fails, is seen to be passing always with any of above combination set using MAVEN_OPTS. Also compared the assembly log as @jonalbrecht mentioned above on qemu setup vs native s390x for `mvn -v` command. The initial few compiled methods match, however it fails for 'java.lang.String::isLatin1': Failure in qemu : ImmutableOopMap{Z_R2=Oop }pc offsets: 170 232 244 272 Compiled method (c1) 1077 12 2 java.lang.String::equalsIgnoreCase (45 bytes) total in heap [0x00000040117f2210,0x00000040117f28b0] = 1696 relocation [0x00000040117f2370,0x00000040117f23c8] = 88 constants [0x00000040117f2400,0x00000040117f2440] = 64 main code [0x00000040117f2440,0x00000040117f2600] = 448 stub code [0x00000040117f2600,0x00000040117f2668] = 104 metadata [0x00000040117f2668,0x00000040117f2688] = 32 scopes data [0x00000040117f2688,0x00000040117f2738] = 176 scopes pcs [0x00000040117f2738,0x00000040117f2888] = 336 dependencies [0x00000040117f2888,0x00000040117f2890] = 8 nul chk table [0x00000040117f2890,0x00000040117f28b0] = 32 ImmutableOopMap{}pc offsets: 288 ImmutableOopMap{Z_R2=Oop Z_R5=Oop }pc offsets: 372 ImmutableOopMap{Z_R5=Oop Z_R2=Oop }pc offsets: 384 392 400 unimplemented opcode 0x0000 # # A fatal error has been detected by the Java Runtime Environment: # # SIGILL (0x4) at pc=0x00000040117f1680, pid=11738, tid=11787 # # JRE version: OpenJDK Runtime Environment (11.0.11+9) (build 11.0.11+9-Ubuntu-0ubuntu2.20.04) # Java VM: OpenJDK 64-Bit Server VM (11.0.11+9-Ubuntu-0ubuntu2.20.04, compiled mode, tiered, compressed oops, g1 gc, linux-s390x) # Problematic frame: # J 9 c1 java.lang.String.hashCode()I java.base (49 bytes) @ 0x00000040117f1680 [0x00000040117f1640+0x0000000000000040] # # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.11738) # # An error report file with more information is saved as: # //hs_err_pid11738.log vs Native s390x log: ImmutableOopMap{Z_R2=Oop }pc offsets: 170 232 244 272 Compiled method (c1) 34 12 2 java.lang.String::equalsIgnoreCase (45 bytes) total in heap [0x000003ff7a097110,0x000003ff7a0977b0] = 1696 relocation [0x000003ff7a097270,0x000003ff7a0972c8] = 88 constants [0x000003ff7a097300,0x000003ff7a097340] = 64 main code [0x000003ff7a097340,0x000003ff7a097500] = 448 stub code [0x000003ff7a097500,0x000003ff7a097568] = 104 metadata [0x000003ff7a097568,0x000003ff7a097588] = 32 scopes data [0x000003ff7a097588,0x000003ff7a097638] = 176 scopes pcs [0x000003ff7a097638,0x000003ff7a097788] = 336 dependencies [0x000003ff7a097788,0x000003ff7a097790] = 8 nul chk table [0x000003ff7a097790,0x000003ff7a0977b0] = 32 ImmutableOopMap{}pc offsets: 276 ImmutableOopMap{Z_R2=Oop Z_R5=Oop }pc offsets: 360 ImmutableOopMap{Z_R5=Oop Z_R2=Oop }pc offsets: 372 380 388 Compiled method (c1) 34 13 2 java.lang.String::isLatin1 (19 bytes) total in heap [0x000003ff7a097810,0x000003ff7a097c10] = 1024 relocation [0x000003ff7a097970,0x000003ff7a097990] = 32 constants [0x000003ff7a0979c0,0x000003ff7a097a00] = 64 main code [0x000003ff7a097a00,0x000003ff7a097b40] = 320 stub code [0x000003ff7a097b40,0x000003ff7a097b98] = 88 metadata [0x000003ff7a097b98,0x000003ff7a097ba0] = 8 scopes data [0x000003ff7a097ba0,0x000003ff7a097bb8] = 24 scopes pcs [0x000003ff7a097bb8,0x000003ff7a097c08] = 80 dependencies [0x000003ff7a097c08,0x000003ff7a097c10] = 8 .................................................. This is an automated cleanup. This bug report has been moved to QEMU's new bug tracker on gitlab.com and thus gets marked as 'expired' now. Please continue with the discussion here: https://gitlab.com/qemu-project/qemu/-/issues/319