Repository navigation
Conversation
|
Scope clarification: issue #17 separately reproduces the W^X failure with Go 1.26. This PR's static backend is gated by |
Add a static Go 1.27 simd/archsimd AVX2 backend under go1.27 && goexperiment.simd so amd64 builds with GOEXPERIMENT=simd on Go 1.27+ encode and decode without runtime executable page mapping (W^X / MemoryDenyWriteExecute=yes) or sonic/loader. - Gate the static backend on go1.27 && goexperiment.simd and guard internal/native (including avx2/sse subpackages and templates) with !go1.27 || !goexperiment.simd so Go <= 1.26 stays on internal/native (preserving _subr__b64decode for sonic's !go1.27 JIT) while Go 1.27+ passes GOEXPERIMENT=simd go test ./... across all root-module packages. - Match native/native.h JSON mode unescaping (\r, \n, \/, ASCII \uXXXX), raw CRLF acceptance, zero-alloc in-place decoding, and src-indexed base64.CorruptInputError offsets. - Document that _subr__b64decode and _subr__b64encode remain 0 under go1.27 && goexperiment.simd, matching -tags noasm and appengine builds. - Place Apache-2.0 license headers before build constraints and ignore generated internal/simd/*.go files in .licenserc.yaml. Part of cloudwego#17
d21f079 to
d7e21c1
Compare
|
@dajiaohuang Thank you for the review — updated the PR title, description, and commit footer from Concretely:
This update also includes three fixes alongside the scope change:
|
Description
Part of #17.
This pull request adds an ahead-of-time SIMD backend for Go 1.27+ on
amd64gated bygo1.27 && goexperiment.simd(simd/archsimd), avoiding runtime executable memory mapping (W^X/MemoryDenyWriteExecute=yes) and the Go 1.27sonic/loadersymbol-table initialization crash when building withGOEXPERIMENT=simd.Scope with Respect to #17
GOEXPERIMENT=simdon Go 1.27+amd64): Static Gosimd/archsimdAVX2 kernels (internal/simd), with automatic fallback toencoding/base64on non-AVX2amd64CPUs. No runtimemmap/mprotect(PROT_EXEC)and nobytedance/sonic/loaderinitialization occur on this path.<= 1.26default builds and Go 1.27 withoutGOEXPERIMENT=simd):<= 1.26((!go1.27 || !goexperiment.simd) && amd64 && !noasm && !appengine),base64xkeeps the existinginternal/native+bytedance/sonic/loaderpath even ifGOEXPERIMENT=simdis set, becausebytedance/sonic(v1.15.0, gated bygo1.17 && !go1.27) links_subr__b64decodevia//go:linknameand emits direct C-ABI calls from its JIT decoder. Under strictW^X(MemoryDenyWriteExecute=yes) on Go<= 1.26, building with-tags noasmworks around the issue by selectingbase64x_fallback.go(and disablingsonic's JIT), while the default-build Go 1.26W^Xcase itself remains open in [BUG] Fatal crash at init under Go 1.27 (runtime.moduledataverify1) and panic under W^X (MemoryDenyWriteExecute) #17.GOEXPERIMENT=simdremain dependent on an upstreambytedance/sonic/loaderupdate for Go 1.27'spcHeader/moduledatalayout.Key Changes
base64x_simd_amd64.go): Conditioned on//go:build go1.27 && goexperiment.simd && amd64 && !noasm && !appengine, dispatching to AVX2simd/archsimdroutines for standard and URL alphabets (padded and raw), CRLF handling, zero-allocation in-place decoding, andnative/native.h-compatible JSON escape unescaping (\r,\n,\/, ASCII\uXXXX) withsrc-indexedbase64.CorruptInputErroroffsets.internal/simd): Ahead-of-time vector encode/decode routines (b64_gen.go) targeting Go 1.27'ssimd/archsimdAPI, with scalar fallback (b64_nosimd_gen.go) and destination capacity guards for 32-byte vector stores.go1.27 && goexperiment.simd: Guardedbase64x_native.go,base64x_subr_native_amd64.go,base64x_error_native_amd64_test.go, andinternal/native(includingavx2,sse, and.tmpltemplates) with!go1.27 || !goexperiment.simdsoGOEXPERIMENT=simd go test -race ./...passes across all packages of the root module on Go 1.27+ without initializingsonic/loader, while Go<= 1.26root-package builds remain oninternal/native._subr__b64decode/_subr__b64encodeSemantics: Documented inbase64x_simd_amd64.goandbase64x_subr_amd64.gothat bothuintptrsymbols remain declared (preserving linkability) and stay0undergo1.27 && goexperiment.simd(wheresonic's!go1.27JIT is inactive), matching-tags noasmandappenginebuilds..licenserc.yaml): Placed Apache-2.0 headers above//go:builddirectives and addedinternal/simd/*.go(b64_gen.go,b64_nosimd_gen.go) topaths-ignorealongsideinternal/native/{avx2,sse}/*.go.Verification
GOEXPERIMENT=simd go test -race -count=3 ./...(Go 1.27.2linux/amd64)go test -tags noasm -race -count=3 ./...andgo test -tags appengine -race -count=1 ./...go test -race ./...andGOEXPERIMENT=simd go test -race . ./internal/native) and Go 1.20.14 (go test -race ./...)linux/arm64(./...) andlinux/riscv64(.)-cpu=1,0 B/op,0 allocs/op, superseding the preliminary [BUG] Fatal crash at init under Go 1.27 (runtime.moduledataverify1) and panic under W^X (MemoryDenyWriteExecute) #17 numbers):GOEXPERIMENT=simd go test -cpu=1 -bench=BenchmarkSimd -benchmem .):BenchmarkSimdEncode_4kB7.72 GB/s,BenchmarkSimdDecode_4kB5.56 GB/s./bench(cd ./bench && GOEXPERIMENT=simd go test -cpu=1 -bench=. -benchmem -run=none ./...):BenchmarkEncoderBase64x_128B4.81 GB/s,4kB7.73 GB/s,256kB7.85 GB/s,1MB7.80 GB/s(vsencoding/base642.27–2.47 GB/s), andBenchmarkDecoderBase64x(64 B input)1.09 GB/s