PID: 0.904
performance: 0.902
other: 0.898
permissions: 0.892
socket: 0.884
vnc: 0.883
graphic: 0.880
semantic: 0.876
debug: 0.875
device: 0.871
network: 0.865
files: 0.860
boot: 0.860
KVM: 0.835

qemu TCG in s390x mode issue with calculating HASH

When using go on s390x on Debian x64 (buster) (host) and debian s390x (sid) (guest) I run into the following problem :

The following occurs while trying to build a custom project :

go: <email address hidden>: Get https://proxy.golang.org/github.com/%21factom%21project/basen/@v/v0.0.0-20150613233007-fe3947df716e.mod: local error: tls: bad record MAC

Doing a git bisect I find that this problem only occurs on and after commit 08ef92d556c584c7faf594ff3af46df456276e1b

Before that commit, all works fine. Past this commit, build always fails.

Without any proof, It looks like a hash calculation bug related to using z/Arch vector facilities...

On 08.10.19 14:11, Cornelia Huck wrote:
> On Tue, 08 Oct 2019 11:19:25 -0000
> Ivan Warren via <email address hidden> wrote:
> 
>> Public bug reported:
>>
>> When using go on s390x on Debian x64 (buster) (host) and debian s390x
>> (sid) (guest) I run into the following problem :
>>
>> The following occurs while trying to build a custom project :
>>
>> go: <email address hidden>:
>> Get
>> https://proxy.golang.org/github.com/%21factom%21project/basen/@v/v0.0.0-20150613233007-fe3947df716e.mod:
>> local error: tls: bad record MAC
>>
>> Doing a git bisect I find that this problem only occurs on and after
>> commit 08ef92d556c584c7faf594ff3af46df456276e1b
>>
>> Before that commit, all works fine. Past this commit, build always
>> fails.
> 
> What version are you using? Current master?
> 
> Can you please share your command line?
> 
>>
>> Without any proof, It looks like a hash calculation bug related to using
>> z/Arch vector facilities...
> 
> Not an unreasonable guess, cc:ing David in case he has seen that before.
> 

Can you reproduce with "-cpu qemu,vx=off" added to the QEMU command
line? Could be some fallout from vector instruction support. Currently
ill, will have a look when I'm feeling better.

-- 

Thanks,

David / dhildenb



On 10/8/2019 3:35 PM, David Hildenbrand wrote:
> On 08.10.19 14:11, Cornelia Huck wrote:
>> On Tue, 08 Oct 2019 11:19:25 -0000
>> Ivan Warren via <email address hidden> wrote:
>>
>>> Public bug reported:
>>>
>>> When using go on s390x on Debian x64 (buster) (host) and debian s390x
>>> (sid) (guest) I run into the following problem :
>>>
>>> The following occurs while trying to build a custom project :
>>>
>>> go: <email address hidden>:
>>> Get
>>> https://proxy.golang.org/github.com/%21factom%21project/basen/@v/v0.0.0-20150613233007-fe3947df716e.mod:
>>> local error: tls: bad record MAC
>>>
>>> Doing a git bisect I find that this problem only occurs on and after
>>> commit 08ef92d556c584c7faf594ff3af46df456276e1b
>>>
>>> Before that commit, all works fine. Past this commit, build always
>>> fails.
>> What version are you using? Current master?
>>
>> Can you please share your command line?
>>
>>> Without any proof, It looks like a hash calculation bug related to using
>>> z/Arch vector facilities...
>> Not an unreasonable guess, cc:ing David in case he has seen that before.
>>
> Can you reproduce with "-cpu qemu,vx=off" added to the QEMU command
> line? Could be some fallout from vector instruction support. Currently
> ill, will have a look when I'm feeling better.

Reposted with a reply all... (sorry for the duplicates)

So it does !


My qemu command line is now (forget the odd funny networking things..)

qemu-system-s390x \
     -drive 
file=DEB002.IMG.NEW,discard=unmap,cache=writeback,id=drive-0,if=none \
     -device virtio-scsi-ccw,id=virtio-scsi-0 \
     -device scsi-hd,id=scsi-hd-0,drive=drive-0 \
     -m 8G \
     -net nic,macaddr=52:54:00:00:00:02 \
     -net tap,ifname=taparm,script=no \
     -nographic -accel tcg,thread=multi \
     -monitor unix:ms,server,nowait \
     -cpu qemu,vx=off \  ##### THAT WAS ADDED as instructed - without it 
everything goes kaput !
     -smp 12

And using the latest bleeding edge qemu from github, my build works (the 
problem goes away).

So the z/Arch vector instructions may have a glitch is a venue to 
consider.. Probably one that couldn't be screened through conventional 
methods.

I'm not that versed into z/Arch vector instruction, but if there 
anything I can help with, I will !

--Ivan





On 08.10.19 16:11, Ivan Warren wrote:
> 
> On 10/8/2019 3:35 PM, David Hildenbrand wrote:
>> On 08.10.19 14:11, Cornelia Huck wrote:
>>> On Tue, 08 Oct 2019 11:19:25 -0000
>>> Ivan Warren via <email address hidden> wrote:
>>>
>>>> Public bug reported:
>>>>
>>>> When using go on s390x on Debian x64 (buster) (host) and debian s390x
>>>> (sid) (guest) I run into the following problem :
>>>>
>>>> The following occurs while trying to build a custom project :
>>>>
>>>> go: <email address hidden>:
>>>> Get
>>>> https://proxy.golang.org/github.com/%21factom%21project/basen/@v/v0.0.0-20150613233007-fe3947df716e.mod:
>>>> local error: tls: bad record MAC
>>>>
>>>> Doing a git bisect I find that this problem only occurs on and after
>>>> commit 08ef92d556c584c7faf594ff3af46df456276e1b
>>>>
>>>> Before that commit, all works fine. Past this commit, build always
>>>> fails.
>>> What version are you using? Current master?
>>>
>>> Can you please share your command line?
>>>
>>>> Without any proof, It looks like a hash calculation bug related to using
>>>> z/Arch vector facilities...
>>> Not an unreasonable guess, cc:ing David in case he has seen that before.
>>>
>> Can you reproduce with "-cpu qemu,vx=off" added to the QEMU command
>> line? Could be some fallout from vector instruction support. Currently
>> ill, will have a look when I'm feeling better.
> 
> Reposted with a reply all... (sorry for the duplicates)
> 
> So it does !
> 
> 
> My qemu command line is now (forget the odd funny networking things..)
> 
> qemu-system-s390x \
>       -drive
> file=DEB002.IMG.NEW,discard=unmap,cache=writeback,id=drive-0,if=none \
>       -device virtio-scsi-ccw,id=virtio-scsi-0 \
>       -device scsi-hd,id=scsi-hd-0,drive=drive-0 \
>       -m 8G \
>       -net nic,macaddr=52:54:00:00:00:02 \
>       -net tap,ifname=taparm,script=no \
>       -nographic -accel tcg,thread=multi \
>       -monitor unix:ms,server,nowait \
>       -cpu qemu,vx=off \  ##### THAT WAS ADDED as instructed - without it
> everything goes kaput !
>       -smp 12
> 
> And using the latest bleeding edge qemu from github, my build works (the
> problem goes away).
> 
> So the z/Arch vector instructions may have a glitch is a venue to
> consider.. Probably one that couldn't be screened through conventional
> methods.
> 
> I'm not that versed into z/Arch vector instruction, but if there
> anything I can help with, I will !

I'll have to reproduce, can you outline the steps needed to trigger 
this? (never had to build a go project before #luckyme ( ;) )). It looks 
like https://github.com/FactomProject/basen is getting pulled in from 
some other project?

Thanks!


-- 

Thanks,

David / dhildenb


On 14.10.19 11:53, David Hildenbrand wrote:
> On 08.10.19 16:11, Ivan Warren wrote:
>>
>> On 10/8/2019 3:35 PM, David Hildenbrand wrote:
>>> On 08.10.19 14:11, Cornelia Huck wrote:
>>>> On Tue, 08 Oct 2019 11:19:25 -0000
>>>> Ivan Warren via <email address hidden> wrote:
>>>>
>>>>> Public bug reported:
>>>>>
>>>>> When using go on s390x on Debian x64 (buster) (host) and debian s390x
>>>>> (sid) (guest) I run into the following problem :
>>>>>
>>>>> The following occurs while trying to build a custom project :
>>>>>
>>>>> go: <email address hidden>:
>>>>> Get
>>>>> https://proxy.golang.org/github.com/%21factom%21project/basen/@v/v0.0.0-20150613233007-fe3947df716e.mod:
>>>>> local error: tls: bad record MAC
>>>>>
>>>>> Doing a git bisect I find that this problem only occurs on and after
>>>>> commit 08ef92d556c584c7faf594ff3af46df456276e1b
>>>>>
>>>>> Before that commit, all works fine. Past this commit, build always
>>>>> fails.
>>>> What version are you using? Current master?
>>>>
>>>> Can you please share your command line?
>>>>
>>>>> Without any proof, It looks like a hash calculation bug related to using
>>>>> z/Arch vector facilities...
>>>> Not an unreasonable guess, cc:ing David in case he has seen that before.
>>>>
>>> Can you reproduce with "-cpu qemu,vx=off" added to the QEMU command
>>> line? Could be some fallout from vector instruction support. Currently
>>> ill, will have a look when I'm feeling better.
>>
>> Reposted with a reply all... (sorry for the duplicates)
>>
>> So it does !
>>
>>
>> My qemu command line is now (forget the odd funny networking things..)
>>
>> qemu-system-s390x \
>>        -drive
>> file=DEB002.IMG.NEW,discard=unmap,cache=writeback,id=drive-0,if=none \
>>        -device virtio-scsi-ccw,id=virtio-scsi-0 \
>>        -device scsi-hd,id=scsi-hd-0,drive=drive-0 \
>>        -m 8G \
>>        -net nic,macaddr=52:54:00:00:00:02 \
>>        -net tap,ifname=taparm,script=no \
>>        -nographic -accel tcg,thread=multi \
>>        -monitor unix:ms,server,nowait \
>>        -cpu qemu,vx=off \  ##### THAT WAS ADDED as instructed - without it
>> everything goes kaput !
>>        -smp 12
>>
>> And using the latest bleeding edge qemu from github, my build works (the
>> problem goes away).
>>
>> So the z/Arch vector instructions may have a glitch is a venue to
>> consider.. Probably one that couldn't be screened through conventional
>> methods.
>>
>> I'm not that versed into z/Arch vector instruction, but if there
>> anything I can help with, I will !
> 
> I'll have to reproduce, can you outline the steps needed to trigger
> this? (never had to build a go project before #luckyme ( ;) )). It looks
> like https://github.com/FactomProject/basen is getting pulled in from
> some other project?
> 

I just tried with Fedora 31 Nightly using "go get"

[root@f31 ~]# go get -v -d github.com/FactomProject/factom
github.com/FactomProject/factom (download)
github.com/FactomProject/btcutil (download)
github.com/FactomProject/ed25519 (download)
github.com/FactomProject/go-bip32 (download)
github.com/FactomProject/btcutilecc (download)
package golang.org/x/crypto/ripemd160: unrecognized import path "golang.org/x/crypto/ripemd160" (https fetch: Get https://golang.org/x/crypto/ripemd160?go-get=1: local error: tls: bad record MAC)
github.com/FactomProject/go-bip39 (download)
package golang.org/x/crypto/pbkdf2: unrecognized import path "golang.org/x/crypto/pbkdf2" (https fetch: Get https://golang.org/x/crypto/pbkdf2?go-get=1: local error: tls: bad record MAC)
github.com/FactomProject/go-bip44 (download)
github.com/FactomProject/netki-go-partner-client (download)
github.com/FactomProject/go-simplejson (download)

With vx=off:

[root@f31 ~]# go get -v -d github.com/FactomProject/factom
github.com/FactomProject/factom (download)
github.com/FactomProject/btcutil (download)
github.com/FactomProject/ed25519 (download)
github.com/FactomProject/go-bip32 (download)
github.com/FactomProject/basen (download)
github.com/FactomProject/btcutilecc (download)
get "golang.org/x/crypto/ripemd160": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at //golang.org/x/crypto/ripemd160?go-get=1
get "golang.org/x/crypto/ripemd160": verifying non-authoritative meta tag
golang.org/x/crypto (download)
github.com/FactomProject/go-bip39 (download)
github.com/FactomProject/go-bip44 (download)
github.com/FactomProject/netki-go-partner-client (download)
github.com/FactomProject/go-simplejson (download)


That should be sufficient to identify the instruction. Might take some time, though. E.g.,
the HASH calculation in the kernel works just fine.

-- 

Thanks,

David / dhildenb


I was looking for a simpler method (without using go), but curl/wget and whatnot don't seem to have any problem..  There must be something go is doing that is different.

My guess (but ONLY a guess) is that the "go" TLS code might be doing some dynamic checking on the z/Arch facility list (STFLE ?) and do some dynamic change to the code path. Maybe a bit far fetched...

Since wget/curl use the openssl library - which (unless a specific engine is specified) uses the lowest common denominator - it will work even when z/Arch VX is present.. However, "golang" may be using a different technique (and may not be using openssl at all) - or invoke openssl with a specific engine if it detects z/Arch VX... 

However, I have the workaround (vx=off), but I'd love to see if there is something amiss with z/Arch VX (for purely educational purpose).

On 14.10.19 12:22, David Hildenbrand wrote:
> On 14.10.19 11:53, David Hildenbrand wrote:
>> On 08.10.19 16:11, Ivan Warren wrote:
>>>
>>> On 10/8/2019 3:35 PM, David Hildenbrand wrote:
>>>> On 08.10.19 14:11, Cornelia Huck wrote:
>>>>> On Tue, 08 Oct 2019 11:19:25 -0000
>>>>> Ivan Warren via <email address hidden> wrote:
>>>>>
>>>>>> Public bug reported:
>>>>>>
>>>>>> When using go on s390x on Debian x64 (buster) (host) and debian s390x
>>>>>> (sid) (guest) I run into the following problem :
>>>>>>
>>>>>> The following occurs while trying to build a custom project :
>>>>>>
>>>>>> go: <email address hidden>:
>>>>>> Get
>>>>>> https://proxy.golang.org/github.com/%21factom%21project/basen/@v/v0.0.0-20150613233007-fe3947df716e.mod:
>>>>>> local error: tls: bad record MAC
>>>>>>
>>>>>> Doing a git bisect I find that this problem only occurs on and after
>>>>>> commit 08ef92d556c584c7faf594ff3af46df456276e1b
>>>>>>
>>>>>> Before that commit, all works fine. Past this commit, build always
>>>>>> fails.
>>>>> What version are you using? Current master?
>>>>>
>>>>> Can you please share your command line?
>>>>>
>>>>>> Without any proof, It looks like a hash calculation bug related to using
>>>>>> z/Arch vector facilities...
>>>>> Not an unreasonable guess, cc:ing David in case he has seen that before.
>>>>>
>>>> Can you reproduce with "-cpu qemu,vx=off" added to the QEMU command
>>>> line? Could be some fallout from vector instruction support. Currently
>>>> ill, will have a look when I'm feeling better.
>>>
>>> Reposted with a reply all... (sorry for the duplicates)
>>>
>>> So it does !
>>>
>>>
>>> My qemu command line is now (forget the odd funny networking things..)
>>>
>>> qemu-system-s390x \
>>>         -drive
>>> file=DEB002.IMG.NEW,discard=unmap,cache=writeback,id=drive-0,if=none \
>>>         -device virtio-scsi-ccw,id=virtio-scsi-0 \
>>>         -device scsi-hd,id=scsi-hd-0,drive=drive-0 \
>>>         -m 8G \
>>>         -net nic,macaddr=52:54:00:00:00:02 \
>>>         -net tap,ifname=taparm,script=no \
>>>         -nographic -accel tcg,thread=multi \
>>>         -monitor unix:ms,server,nowait \
>>>         -cpu qemu,vx=off \  ##### THAT WAS ADDED as instructed - without it
>>> everything goes kaput !
>>>         -smp 12
>>>
>>> And using the latest bleeding edge qemu from github, my build works (the
>>> problem goes away).
>>>
>>> So the z/Arch vector instructions may have a glitch is a venue to
>>> consider.. Probably one that couldn't be screened through conventional
>>> methods.
>>>
>>> I'm not that versed into z/Arch vector instruction, but if there
>>> anything I can help with, I will !
>>
>> I'll have to reproduce, can you outline the steps needed to trigger
>> this? (never had to build a go project before #luckyme ( ;) )). It looks
>> like https://github.com/FactomProject/basen is getting pulled in from
>> some other project?
>>
> 
> I just tried with Fedora 31 Nightly using "go get"
> 
> [root@f31 ~]# go get -v -d github.com/FactomProject/factom
> github.com/FactomProject/factom (download)
> github.com/FactomProject/btcutil (download)
> github.com/FactomProject/ed25519 (download)
> github.com/FactomProject/go-bip32 (download)
> github.com/FactomProject/btcutilecc (download)
> package golang.org/x/crypto/ripemd160: unrecognized import path "golang.org/x/crypto/ripemd160" (https fetch: Get https://golang.org/x/crypto/ripemd160?go-get=1: local error: tls: bad record MAC)
> github.com/FactomProject/go-bip39 (download)
> package golang.org/x/crypto/pbkdf2: unrecognized import path "golang.org/x/crypto/pbkdf2" (https fetch: Get https://golang.org/x/crypto/pbkdf2?go-get=1: local error: tls: bad record MAC)
> github.com/FactomProject/go-bip44 (download)
> github.com/FactomProject/netki-go-partner-client (download)
> github.com/FactomProject/go-simplejson (download)
> 
> With vx=off:
> 
> [root@f31 ~]# go get -v -d github.com/FactomProject/factom
> github.com/FactomProject/factom (download)
> github.com/FactomProject/btcutil (download)
> github.com/FactomProject/ed25519 (download)
> github.com/FactomProject/go-bip32 (download)
> github.com/FactomProject/basen (download)
> github.com/FactomProject/btcutilecc (download)
> get "golang.org/x/crypto/ripemd160": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at //golang.org/x/crypto/ripemd160?go-get=1
> get "golang.org/x/crypto/ripemd160": verifying non-authoritative meta tag
> golang.org/x/crypto (download)
> github.com/FactomProject/go-bip39 (download)
> github.com/FactomProject/go-bip44 (download)
> github.com/FactomProject/netki-go-partner-client (download)
> github.com/FactomProject/go-simplejson (download)
> 
> 
> That should be sufficient to identify the instruction. Might take some time, though. E.g.,
> the HASH calculation in the kernel works just fine.
> 

By now I am pretty sure the code that gets triggered is

src/vendor/golang.org/x/crypto/poly1305/sum_s390x.s

in the go repository.

I started writing unit tests for all involved vector instructions, no 
luck so far. Could also be some side-effect/BUG of another instruction.

Will let you know once I know more :)

-- 

Thanks,

David / dhildenb


Looks fixed to me ! The issue no longer shows even without specifying vx=off

On 23.10.19 11:37, Ivan Warren wrote:
> Looks fixed to me ! The issue no longer shows even without specifying
> vx=off
> 

Nice, I suspect that there might be more issues when using golang (as it 
really makes excessive use of vector registers to my surprise). So in 
case you run into problems (and especially can't reproduce with vx=off), 
please inform me!

Thanks!

-- 

Thanks,

David / dhildenb



I will exercise this thoroughly ! The go application involved is itself a blockchain signing/verification application, so I suspect it will be a good exercise (codenotary.io)

Thanks for looking into this and fixing this !

--Ivan

I assume David's fixes have been released with QEMU v4.2, so I'm closing this ticket now.

Thank you so much!  I've been hitting the same issue with s390x.

Maybe an endian issue with hardware vx?

Also affects RISC-V apparently. https://github.com/moby/moby/issues/40240

Tried applying the -cpu parameter into Qemu 4.1.1 but I got:

qemu-system-riscv64: unable to find CPU model 'qemu'

My command is:

qemu-system-riscv64 \
    -machine virt \
    -cpu qemu,vx=off \
    -nographic \
    -smp 4 \
    -m 4G \
    -bios fw_payload.bin \
    -device virtio-blk-device,drive=hd0 \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-device,rng=rng0 \
    -drive file=riscv64-debianrootfs-qemu.qcow2,format=qcow2,id=hd0 \
    -device virtio-net-device,netdev=usernet \
    -netdev user,id=usernet,hostfwd=tcp::22222-:22"