cmd/go/internal/work: wrong arch detection for s390x

[taronaeo]

Go version

go version devel go1.25-5eaeb7b455 Thu Apr 3 15:36:36 2025 -0700 linux/s390x

Output of go env in your module/workspace:

AR='ar'                                                                                                                             
CC='gcc'                                                                                                                            
CGO_CFLAGS='-O2 -g'                                                                                                                 
CGO_CPPFLAGS=''                                                                                                                     
CGO_CXXFLAGS='-O2 -g'                                                                                                               
CGO_ENABLED='1'                                                                                                                     
CGO_FFLAGS='-O2 -g'                                                                                                                 
CGO_LDFLAGS='-O2 -g'                                                                                                                
CXX='g++'                                                                                                                           
GCCGO='/usr/bin/gccgo'                                                                                                              
GO111MODULE=''                                                                                                                      
GOARCH='s390x'                                                                                                                      
GOAUTH='netrc'                                                                                                                      
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -march=z196 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1226758756=/tmp/go-build -gno-record-gcc-switches' 
GOHOSTARCH='s390x'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/opt/ollama/go.mod'
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/goroot'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/goroot/pkg/tool/linux_s390x'
GOVCS=''
GOVERSION='devel go1.25-5eaeb7b455 Thu Apr 3 15:36:36 2025 -0700'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I'm trying to compile Ollama with -march=z15 -mtune=z15 -mvx -mzvector but runtime/cgo is reporting that hardware vector support is not available on z196 while my actual machine is z15.

$ uname -a
Linux aitest 5.14.0-575.el9.s390x #1 SMP Tue Mar 25 17:28:06 UTC 2025 s390x s390x s390x GNU/Linux

$ CGO_CPPFLAGS="-I/opt/openblas-libs/include" CGO_LDFLAGS="-L/opt/openblas-libs/lib -lopenblas" CC="gcc -march=z15 -mtune=z15 -mvx -mzvector" go run . serve

# runtime/cgo
cc1: error: hardware vector support not available on z196

Running a check on the environment variables reported that the machine type is indeed enforced as z196, which is incorrect.

$ go env | grep GOGCCFLAGS
GOGCCFLAGS='-fPIC -m64 -march=z196 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2459717027=/tmp/go-build -gno-record-gcc-switches'

Trying to override the GOGCCFLAGS was met with go: GOGCCFLAGS cannot be modified as follows

$ go env -w GOGCCFLAGS="-fPIC -m64 -march=z15 -mtune=z15 -mvx -mzvector -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2459717027=/tmp/go-build -gno-record-gcc-switches"
go: GOGCCFLAGS cannot be modified

What did you see happen?

My Ollama development project fails to compile with -march=z15 -mtune=z15 -mvx -mzvector because the machine type is enforced to z196 which is incorrect.

What did you expect to see?

I was expecting Go to use the correct machine type i.e., z15 = -march=z15, z16 = -march=z16 instead of defaulting to -march=z196 which is very old. Otherwise, a possible method of overriding the GOGCCFLAGS to fix this bug.

[gabyhelp]

Related Issues

• x/image/vector TestMakeFlInXxx fails on s390x and ppc64le #21460 (closed)
• x/crypto/sha3: When compiling for all platforms `undefined: cpu.S390X` #31227 (closed)
• cmd/compile: illegal instruction on s390x #16637 (closed)
• cmd/go: TestCgoConsistentResults fails on s390x with gcc 6 #16780 (closed)
• cmd/compile: linux/s390x: inlining bug in s390x #64468 (closed)

Related Code Changes

• runtime: handle SIGPE exception for p256 crypto package on s390x

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

I believe it comes from https://go.googlesource.com/go/+/ba7b8ca336123017e43a2ab3310fd4a82122ef9d/src/cmd/go/internal/work/exec.go#2590

[taronaeo]

Hi @seankhliao, thanks for quickly triaging this issue! Yep, it appears to be coming from there.

By any chance can we add conditional detection, otherwise default to -march=native instead of using -march=z196?

Edit: I think with conditional detection, we should also be able to turn on specific hardware support i.e., -mvx -mzvector for z15. Maybe a reference to src/runtime/os_linux_s390x.go?

[taronaeo]

I have temporarily patched Golang on my side and have verified that the referenced file is causing the problem. Please let me know if the Golang team is looking into a patch or if I should try and make one on my own, and create a PR :)

[mknyszek]

CC @golang/s390x

[mknyszek]

I think a patch would be welcome, but it would be good to get feedback from the s390x maintainers.

[srinivas-pokala]

@taronaeo thank's for raising the issue.
In s390x, for CGO we support z196 onwards, which we can see using go env GOGCCFLAGS as below
GOGCCFLAGS='-fPIC -m64 -march=z196 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1600764568=/tmp/go-build -gno-record-gcc-switches'
But when we try to override gcc to specific machine version and capabilities using CC="gcc -march=z15 -mtune=z15 -mvx -mzvector" go run <*.go , it concatinates and this with default machine details Default_machine.

Due to this later argument(-march=z196) overrides the earlier one(-march=z15) and exception happening cc1: error: hardware vector support not available on z196 . I have verified this on s390x machine and I will raise CL for the same.

@mknyszek

[gopherbot]

Change https://go.dev/cl/666995 mentions this issue: cmd/go/internal/work: fix arch detection for s390x

[srinivas-pokala]

@taronaeo you can use cflags(CGO_CFLAGS) which is standard practice in many of the projects.

CGO_CPPFLAGS="-I/opt/openblas-libs/include" CGO_LDFLAGS="-L/opt/openblas-libs/lib -lopenblas" CGO_CFLAGS="-march=z15 -mtune=z15 -mvx -mzvector" go run . serve
Can you try with change in the command using CGO_CFLAGS

[taronaeo]

Hi @srinivas-pokala, thanks for verifying the problem on your end. I can reproduce the successful compilation using the suggested CGO_CFLAGS environment variable. However, this solution is only a temporary fix for the issue, as anyone on the s390x platform should be able to build the same project on a different machine without needing to include additional environment variables.

I noticed this in the platform detection code,

1. Unlike architectures such as AMD64 and ARM64, IBM Z & LinuxONE mainframes follow an incremental -march approach where, for example, z15 would get its own -march=z15 flag and z16 would get -march=z16, etc. But this isn't the same in AMD64, where it would only require one -march=x86_64 flag.
   a. To fix this on s390x, we would either need to use -march=native and allow GCC/Clang to detect the platform natively, or;
   b. Detect the platform ourselves and enable the relevant hardware instruction sets either within Golang or the project

Feel free to correct me if I'm wrong. In my opinion, the easiest way is to change -march=z196 to -march=native and let the project enable the necessary hardware instruction set via #cgo within its code. This would quickly fix the compilation issues with hardware instruction sets enabled, and also allow anyone to build without needing to include additional environment variables.

Apart from the start of support (mentioned here), was there a documented reason why the Golang team chose -march=z196 instead of -march=native?

Documentation for -march=native: https://gcc.gnu.org/onlinedocs/gcc/S_002f390-and-zSeries-Options.html#index-march-14