path: root/results/scraper/launchpad/1920913

Openjdk11+ fails to install on s390x

While installing openjdk11 or higher from repo, it crashes while configuring ca-certificates-java.
Although `java -version` passes, `jar -version` crashes. Detailed logs attached to this issue.

```
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x00000040126f9980, pid=8425, tid=8430
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 4 c1 java.lang.StringLatin1.hashCode([B)I java.base@11.0.10 (42 bytes) @ 0x00000040126f9980 [0x00000040126f9980+0x0000000000000000]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.8425)
#
# An error report file with more information is saved as:
# //hs_err_pid8425.log
sed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /root/core.10740)
#
# An error report file with more information is saved as:
# /root/hs_err_pid10740.log
```

Observed this on s390x/ubuntu as well as alpine when run on amd64. 
Please note, on native s390x, the installation is successful. Also this crash is not observed while installing openjdk-8-jdk. 

Qemu version: 5.2.0

Please let me know if any more details are needed.



You don't say how you're invoking QEMU (system emulation? usermode? what command line?) Please give the full commandline, repro steps, and any files/images we would need to reproduce the failure.


Please find below steps to reproduce the issue(Running on amd64 VM):

```
apt-get install -y qemu qemu-user-static
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker run -it s390x/ubuntu:20.04 bash
--> apt-get update && apt-get install -y openjdk-11-jdk
    jar --version
```



Same BUG as https://bugs.launchpad.net/qemu/+bug/1862874

Tried building jdk 11 from source, the generated executable still crashes(fastdebug as well as release mode):

```
root@24d396a17e00:~/jdk# build/linux-s390x-normal-server-release/jdk/bin/java -version
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x000000400b234440, pid=18175, tid=18178
#
# JRE version: OpenJDK Runtime Environment (11.0) (build 11-internal+0-adhoc..jdk)
# Java VM: OpenJDK 64-Bit Server VM (11-internal+0-adhoc..jdk, mixed mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 78 c1 java.util.HashMap.afterNodeInsertion(Z)V java.base (1 bytes) @ 0x000000400b234440 [0x000000400b234400+0x0000000000000040]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /root/jdk/core.18175)
#
# An error report file with more information is saved as:
# /root/jdk/hs_err_pid18175.log
Compiled method (c1)    1795   78       3       java.util.HashMap::afterNodeInsertion (1 bytes)
 total in heap  [0x000000400b234210,0x000000400b2345b0] = 928
 relocation     [0x000000400b234378,0x000000400b2343a0] = 40
 constants      [0x000000400b2343c0,0x000000400b234400] = 64
 main code      [0x000000400b234400,0x000000400b234500] = 256
 stub code      [0x000000400b234500,0x000000400b234558] = 88
 metadata       [0x000000400b234558,0x000000400b234568] = 16
 scopes data    [0x000000400b234568,0x000000400b234578] = 16
 scopes pcs     [0x000000400b234578,0x000000400b2345a8] = 48
 dependencies   [0x000000400b2345a8,0x000000400b2345b0] = 8
Compiled method (c1)    1806   74       3       java.util.HashMap::putVal (300 bytes)
 total in heap  [0x000000400b230210,0x000000400b231f20] = 7440
 relocation     [0x000000400b230378,0x000000400b230690] = 792
 constants      [0x000000400b2306c0,0x000000400b230a00] = 832
 main code      [0x000000400b230a00,0x000000400b231980] = 3968
 stub code      [0x000000400b231980,0x000000400b231a68] = 232
 metadata       [0x000000400b231a68,0x000000400b231ad0] = 104
 scopes data    [0x000000400b231ad0,0x000000400b231ce8] = 536
 scopes pcs     [0x000000400b231ce8,0x000000400b231eb8] = 464
 dependencies   [0x000000400b231eb8,0x000000400b231ec0] = 8
 nul chk table  [0x000000400b231ec0,0x000000400b231f20] = 96
Could not load hsdis-s390x.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)
root@24d396a17e00:~/jdk#
```

@davidhildenbrand The other issue which you have mentioned as duplicate shows java getting stuck for long, whereas for me it crashes right away. Do you think these 2 are related?

Also observed another behaviour :
java -version randomly passes, sometimes.

I can also confirm that it is observed under s390x chroot as well(logs below):
``` 
root@XX:/# ulimit -c unlimited
root@XX:/# java -version
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode)
root@XX:/# java -version
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x0000004012705b40, pid=156601, tid=156604
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 5 c1 java.lang.Object.<init>()V java.base@11.0.10 (1 bytes) @ 0x0000004012705b40 [0x0000004012705b00+0x0000000000000040]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.156601)
#
# An error report file with more information is saved as:
# //hs_err_pid156601.log
Compiled method (c1)     956    5       3       java.lang.Object::<init> (1 bytes)
 total in heap  [0x0000004012705910,0x0000004012705cb8] = 936
 relocation     [0x0000004012705a70,0x0000004012705aa0] = 48
 constants      [0x0000004012705ac0,0x0000004012705b00] = 64
 main code      [0x0000004012705b00,0x0000004012705c00] = 256
 stub code      [0x0000004012705c00,0x0000004012705c58] = 88
 metadata       [0x0000004012705c58,0x0000004012705c70] = 24
 scopes data    [0x0000004012705c70,0x0000004012705c80] = 16
 scopes pcs     [0x0000004012705c80,0x0000004012705cb0] = 48
 dependencies   [0x0000004012705cb0,0x0000004012705cb8] = 8
Compiled method (c1)     960    5       3       java.lang.Object::<init> (1 bytes)
 total in heap  [0x0000004012705910,0x0000004012705cb8] = 936
 relocation     [0x0000004012705a70,0x0000004012705aa0] = 48
 constants      [0x0000004012705ac0,0x0000004012705b00] = 64
 main code      [0x0000004012705b00,0x0000004012705c00] = 256
 stub code      [0x0000004012705c00,0x0000004012705c58] = 88
 metadata       [0x0000004012705c58,0x0000004012705c70] = 24
 scopes data    [0x0000004012705c70,0x0000004012705c80] = 16
 scopes pcs     [0x0000004012705c80,0x0000004012705cb0] = 48
 dependencies   [0x0000004012705cb0,0x0000004012705cb8] = 8
Could not load hsdis-s390x.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
#
Aborted (core dumped)
root@XX:/# ulimit -c unlimited
root@XX:/# java -version
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x0000004012706a40, pid=156619, tid=156622
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 4 c1 java.lang.Object.<init>()V java.base@11.0.10 (1 bytes) @ 0x0000004012706a40 [0x0000004012706a00+0x0000000000000040]
#
.
(truncating logs)

Aborted (core dumped)
root@XX:/#
```

Increasing core limit worked once, but it fails eventually.

Could you please share your thoughts and provide some pointers on debugging further?

 


As java -version passes few times, further also checked behaviour of Maven. Observed that mvn -v crashes in a similar fashion, however after setting below:
export MAVEN_OPTS="-XX:-TieredCompilation -XX:+UseG1GC -Dcount=1000000"

mvn -v always passes.

root@XX:/# mvn -v
OpenJDK 64-Bit Server VM warning: You have loaded library /apache-maven-3.6.3/lib/jansi-native/linux64/libjansi.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /apache-maven-3.6.3
Java version: 11.0.7, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-s390x
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "5.4.0-70-generic", arch: "s390x", family: "unix"


However what I am really interested in, is mvn clean install command which never passes with above settings.

@davidhildenbrand, any help would be appreciated.


Hi @davidhildenbrand, I'm on the same team as @nam121 and I've been looking at this issue as well.

I think this is the same issue as: https://github.com/multiarch/qemu-user-static/issues/129

I've been running an s390x docker image on a master build (with latest s390x commit from Apr 23) of user mode qemu-s390x-static with some debug logging on: 

$ sudo docker run -e QEMU_CPU="qemu" -e QEMU_LOG="unimp,guest_errors" -e QEMU_LOG_FILENAME="/s390x/qemu_s390x.log"

I ran a simple java program with:

$ java -Xcomp -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:PrintAssemblyOptions=hsdis-print-bytes -XX:+LogCompilation -XX:LogFile=java_compilation_log.log  Main > java_out.txt

and the qemu log contained just one line: 

unimplemented opcode 0x0000

Note that if the JIT is turned off with 'java -Xint', then all programs I've tried run without problem.

The hs_err file reports a SIGILL in the same spot as in the other comments:

--- SNIP
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x00000040126d7680, pid=208, tid=211
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, compiled mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 9 c1 java.lang.String.hashCode()I java.base (49 bytes) @ 0x00000040126d7680 [0x00000040126d7640+0x0000000000000040]
--- SNIP
--- SNIP
Instructions: (pc=0x00000040126d7680)
0x00000040126d7580:   00000040 5f5f4140 00000040 5f5f4140
0x00000040126d7590:   00000040 5f5f4140 00000040 5f5f4140
0x00000040126d75a0:   00000040 5f5f4358 00000040 5f5f4358
0x00000040126d75b0:   00000040 5f5f4358 00000040 5f5f4358
0x00000040126d75c0:   00000040 5f5f4140 00000040 5f5f4140
0x00000040126d75d0:   00000000 00000000 ffffffff ffffffff
0x00000040126d75e0:   00000040 5f5f4140 00000000 00000000
0x00000040126d75f0:   ffffffff ffffffff 00000040 5f3fb9d0
0x00000040126d7600:   00000040 12238c00 00000040 12232800
0x00000040126d7610:   00000040 5f3fef18 00000040 12238c00
0x00000040126d7620:   00000040 12235000 00000000 00000000
0x00000040126d7630:   00000000 00000000 00000000 00000000
0x00000040126d7640:   b9040009 cc08ffff fff85500 2008a784  # <-- String.hashCode() entry point at 0x00000040126d7640
0x00000040126d7650:   0019a51d 0040c019 12167a80 07f10700
0x00000040126d7660:   07000700 07000700 07000700 07000700
0x00000040126d7670:   07000700 07000700 07000700 07000700
0x00000040126d7680:   0000f000 ec51e3e0 f0080024 b904000f  # <-- note 0x0000 at 0x00000040126d7680
0x00000040126d7690:   a7fbffa0 e300f000 0024c438 ffffff73
--- SNIP

The assembly printed by java looks like:

--- SNIP
[Entry Point]
  # {method} {0x000000405f3fb9d0} 'hashCode' '()I' in 'java/lang/String'
  #           [sp+0x60]  (sp of caller)
  0x00000040126d7640: lgr	%r0,%r9         ;...b9040009
                                                ;   {no_reloc}
  0x00000040126d7644: aih	%r0,-8          ;...cc08ffff fff8

  0x00000040126d764a: cl	%r0,8(%r2)      ;...55002008

  0x00000040126d764e: je	0x00000040126d7680  ;...a7840019

  0x00000040126d7652: llihl	%r1,64          ;...a51d0040

  0x00000040126d7656: iilf	%r1,303463040   ;...c0191216 7a80

  0x00000040126d765c: br	%r1             ;...07f1

  0x00000040126d765e: nopr                      ;...0700

  0x00000040126d7660: nopr                      ;...0700

  0x00000040126d7662: nopr                      ;...0700

  0x00000040126d7664: nopr                      ;...0700

  0x00000040126d7666: nopr                      ;...0700

  0x00000040126d7668: nopr                      ;...0700

  0x00000040126d766a: nopr                      ;...0700

  0x00000040126d766c: nopr                      ;...0700

  0x00000040126d766e: nopr                      ;...0700

  0x00000040126d7670: nopr                      ;...0700

  0x00000040126d7672: nopr                      ;...0700

  0x00000040126d7674: nopr                      ;...0700

  0x00000040126d7676: nopr                      ;...0700

  0x00000040126d7678: nopr                      ;...0700

  0x00000040126d767a: nopr                      ;...0700

  0x00000040126d767c: nopr                      ;...0700

  0x00000040126d767e: nopr                      ;...0700

[Verified Entry Point]
  0x00000040126d7680: tmy	-81920(%r15),222  ;...ebdef000 ec51

  0x00000040126d7686: stg	%r14,8(%r15)    ;...e3e0f008 0024

  0x00000040126d768c: lgr	%r0,%r15        ;...b904000f

  0x00000040126d7690: aghi	%r15,-96        ;...a7fbffa0

  0x00000040126d7694: stg	%r0,0(%r15)     ;...e300f000 0024

  0x00000040126d769a: lgrl	%r3,0x00000040126d7580
                                                ;...c438ffff ff73
                                                ;   {metadata(method data for {method} {0x000000405f3fb9d0} 'hashCode' '()I' in 'java/lang/String')}
--- SNIP

so IIUC java says its generating 0xebde at 0x00000040126d7680 instead of 0x0000.

Hope the above makes sense. I'm not sure where to go from here so any suggestions would be a great help.

From looking at the in_asm logs, it looks like that instruction starting with 0xebde is executed once with no problem but the second time its changed to 0x0000.

... # First Time
----------------
IN:
0x40126d6880:  ebde f000 ec51  tmy      -0x14000(%r15), 0xde
0x40126d6886:  e3e0 f008 0024  stg      %r14, 8(%r15)
0x40126d688c:  b904 000f       lgr      %r0, %r15
0x40126d6890:  a7fb ffa0       aghi     %r15, -0x60
0x40126d6894:  e300 f000 0024  stg      %r0, 0(%r15)
0x40126d689a:  c438 ffff ff73  lgrl     %r3, 0x40126d6780
0x40126d68a0:  5840 30dc       l        %r4, 0xdc(%r3)
0x40126d68a4:  c248 0000 0008  agfi     %r4, 8
0x40126d68aa:  5040 30dc       st       %r4, 0xdc(%r3)
0x40126d68ae:  c0f4 0000 00d1  jg       0x40126d6a50
PSW=mask 0000000180000000 addr 00000040126d6880 cc  CC_OP_LTGT0_64
R00=0000000000000000 R01=00000040126d6880 R02=00000006296f5d20 R03=00000006296f5d20
R04=000000405f45fcd8 R05=00000006000000e8 R06=0000004012169380 R07=0000004002c410e8
R08=0000004004019000 R09=000000405f2d29d0 R10=0000004002c41048 R11=00000006296095e0
R12=000000400280ec50 R13=0000004002c411d0 R14=00000040126d5c64 R15=0000004002c40e88

... # Second Time
unimplemented opcode 0x0000
----------------
IN:
PSW=mask 0000000180000000 addr 00000040126d6880 cc CC_OP_LTUGTU_32
R00=0000000000001808 R01=00000040126d53c0 R02=00000006296f5d78 R03=00000006296f5d78
R04=000000405f45fcd8 R05=00000006000000f0 R06=0000004012114000 R07=5f9dbb3700003030
R08=0000004004019000 R09=0000000800001808 R10=0000004002c41048 R11=00000006296095e0
R12=000000400280ec50 R13=0000004002c411d0 R14=00000040126d5c64 R15=0000004002c40e88

Some more analysis:
Tried to explicitely compile as well as exclude few methods during compilation such as  'java.lang.StringLatin1::hashCode', 'java.util.concurrent.ConcurrentHashMap', 'java.lang.String*' which are part of trace as logged in above comments, with the help of advanced JIT options. 
However it is not good enough to draw any conclusion as `java -version` command passes on random runs. `mvn -v` which consistently fails, is seen to be passing always with any of above combination set using MAVEN_OPTS. 

Also compared the assembly log as @jonalbrecht mentioned above on qemu setup vs native s390x for `mvn -v` command. 
The initial few compiled methods match, however it fails for 'java.lang.String::isLatin1':

Failure in qemu :
ImmutableOopMap{Z_R2=Oop }pc offsets: 170 232 244 272 Compiled method (c1)    1077   12       2       java.lang.String::equalsIgnoreCase (45 bytes)
 total in heap  [0x00000040117f2210,0x00000040117f28b0] = 1696
 relocation     [0x00000040117f2370,0x00000040117f23c8] = 88
 constants      [0x00000040117f2400,0x00000040117f2440] = 64
 main code      [0x00000040117f2440,0x00000040117f2600] = 448
 stub code      [0x00000040117f2600,0x00000040117f2668] = 104
 metadata       [0x00000040117f2668,0x00000040117f2688] = 32
 scopes data    [0x00000040117f2688,0x00000040117f2738] = 176
 scopes pcs     [0x00000040117f2738,0x00000040117f2888] = 336
 dependencies   [0x00000040117f2888,0x00000040117f2890] = 8
 nul chk table  [0x00000040117f2890,0x00000040117f28b0] = 32

ImmutableOopMap{}pc offsets: 288
ImmutableOopMap{Z_R2=Oop Z_R5=Oop }pc offsets: 372
ImmutableOopMap{Z_R5=Oop Z_R2=Oop }pc offsets: 384 392 400 unimplemented opcode 0x0000
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x00000040117f1680, pid=11738, tid=11787
#
# JRE version: OpenJDK Runtime Environment (11.0.11+9) (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.11+9-Ubuntu-0ubuntu2.20.04, compiled mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 9 c1 java.lang.String.hashCode()I java.base (49 bytes) @ 0x00000040117f1680 [0x00000040117f1640+0x0000000000000040]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.11738)
#
# An error report file with more information is saved as:
# //hs_err_pid11738.log


vs

Native s390x log:
ImmutableOopMap{Z_R2=Oop }pc offsets: 170 232 244 272 Compiled method (c1)      34   12       2       java.lang.String::equalsIgnoreCase (45 bytes)
 total in heap  [0x000003ff7a097110,0x000003ff7a0977b0] = 1696
 relocation     [0x000003ff7a097270,0x000003ff7a0972c8] = 88
 constants      [0x000003ff7a097300,0x000003ff7a097340] = 64
 main code      [0x000003ff7a097340,0x000003ff7a097500] = 448
 stub code      [0x000003ff7a097500,0x000003ff7a097568] = 104
 metadata       [0x000003ff7a097568,0x000003ff7a097588] = 32
 scopes data    [0x000003ff7a097588,0x000003ff7a097638] = 176
 scopes pcs     [0x000003ff7a097638,0x000003ff7a097788] = 336
 dependencies   [0x000003ff7a097788,0x000003ff7a097790] = 8
 nul chk table  [0x000003ff7a097790,0x000003ff7a0977b0] = 32

ImmutableOopMap{}pc offsets: 276
ImmutableOopMap{Z_R2=Oop Z_R5=Oop }pc offsets: 360
ImmutableOopMap{Z_R5=Oop Z_R2=Oop }pc offsets: 372 380 388 Compiled method (c1)      34   13       2       java.lang.String::isLatin1 (19 bytes)
 total in heap  [0x000003ff7a097810,0x000003ff7a097c10] = 1024
 relocation     [0x000003ff7a097970,0x000003ff7a097990] = 32
 constants      [0x000003ff7a0979c0,0x000003ff7a097a00] = 64
 main code      [0x000003ff7a097a00,0x000003ff7a097b40] = 320
 stub code      [0x000003ff7a097b40,0x000003ff7a097b98] = 88
 metadata       [0x000003ff7a097b98,0x000003ff7a097ba0] = 8
 scopes data    [0x000003ff7a097ba0,0x000003ff7a097bb8] = 24
 scopes pcs     [0x000003ff7a097bb8,0x000003ff7a097c08] = 80
 dependencies   [0x000003ff7a097c08,0x000003ff7a097c10] = 8 

..................................................


This is an automated cleanup. This bug report has been moved to QEMU's
new bug tracker on gitlab.com and thus gets marked as 'expired' now.
Please continue with the discussion here:

 https://gitlab.com/qemu-project/qemu/-/issues/319