hmac_digest race condition with openssl 3.0.19 and Python 3.14t

[lunixbochs]

Crash report

What happened?

summary

I'm seeing a null pointer deref happen often in this stack, after hmac.digest(...) calls in a thread in my app during early startup. I have python 3.14t and openssl 3.0.19 LTS (with python's internal _hmac backend disabled because I can't statically link it). The crash does not happen if I use hmac.new(...).digest(), which takes a slightly different path through openssl. I will note python's _ssl module may also be able to trigger this.

This is happening in a complex app, I haven't been able to find an easy standalone repro.

0   libsystem_pthread.dylib       	0x00007fff2037b714 pthread_rwlock_rdlock + 1
1   libcrypto.3.dylib             	0x00000001022a90f8 ossl_lib_ctx_get_data + 72
2   libcrypto.3.dylib             	0x000000010227b6ce 0x102174000 + 1078990
3   libcrypto.3.dylib             	0x000000010227b089 0x102174000 + 1077385
4   libcrypto.3.dylib             	0x000000010227b02a evp_generic_fetch + 106
5   libcrypto.3.dylib             	0x000000010228c6b3 EVP_MAC_fetch + 67
6   libcrypto.3.dylib             	0x000000010228c363 EVP_Q_mac + 99
7   libcrypto.3.dylib             	0x00000001022a235d HMAC + 253

I have also tested with openssl 3.5.x LTS, and the default libctx issue appears to be fixed in openssl there as they've reworked the default libctx init (they have more explicit init guards).

I don't know what to actually do about this. Maybe the solution is a build-time check against pairing free-threaded Python with pre-3.5 openssl? I also haven't checked the other compatible ssl impls (e.g. boringssl and libressl) to see if they have the same threading issues or if they've fixed them independently.

more detail

If you're using openssl as your hmac backend, _hashopenssl.hmac_digest() can race something inside openssl and crash if you have threads doing hmac during early Python startup. I think this is due to hmac_digest using the implicit default_context_int libctx inside openssl, which doesn't seem thread safe on startup in 3.0.19.
At _hashopenssl.c:66, we see PY_EVP_MD_fetch is defined to use EVP_MD_fetch(NULL, ...), which selects the default libctx.

If there are other parts of cpython using the implicit openssl libctx rather than creating their own context, I suspect those are also not safe with openssl 3.0.19. It looks like the next openssl LTS (3.5.x) may have a better locking mechanism around the default libctx.

It looks like _ssl also uses the default openssl libctx: _ssl.c:3449 - SSL_CTX_new(method) implies the default libctx (where SSL_new_ex() allows you to specify libctx)

CPython versions tested on:

3.14

Operating systems tested on:

macOS

Output from running 'python -VV' on the command line:

No response

[picnixz]

If the problem is in OpenSSL 3.0 and has been fixed later I'm not entirely sure we can do something on our side.

cc @ZeroIntensity

[lunixbochs]

3.0.x is one of the two maintained openssl LTS branches. If there's a possible build level fix to warn people about the combination, it might save someone else a lot of trouble in figuring out why their build is crashing. Otherwise, I think leaving this issue open until September and closing it then would be fine. In my case, I have a workaround, but I want to save other people the few days of trouble this caused me. Also this is a sign it's possible it's worth someone auditing cpython's other native dependencies for shared global state prone to startup race conditions.

[picnixz]

Oh 3.0.x is still LTS, ok. I'm not entirely sure how we can add locks for that as it would create an inter-module dependency. We don't use specific libssl contexts but we might need to do that in order to avoid this kind of race. I haven't digged into it yet.