Repository navigation
Conversation
|
This locally reproduces the X509 failure that flakes in CI. Eight workers decode an RSA-PSS certificate concurrently with background CA loading. On main I get Source |
|
Review requested:
|
Codecov Report❌ Patch coverage is
Additional details and impacted files
🚀 New features to boost your workflow:
|
|
I'll let CQ let me know when 48 hours are up (it'll fail on the account of the revert commit message), then i'll land this manually. |
Commit Queue failedThe pull request was removed from the Commit Queue and labeled
commit-queue-failed
Full Commit Queue output |
|
Landed in 03cca65...6511fb8 |
OpenSSL 3 builds its table of algorithm names on first use. When 43d3fe9 moved that first use after other threads had started, an RSA-PSS key decoded while the table was being built could fail with `ERR_OSSL_EVP_DECODE_ERROR`, and every later RSA-PSS decode in the process failed too. #66903 reverted that change. The new test fails if it comes back: Workers decode an RSA-PSS certificate while the root certificate loader or a thread pool crypto job uses OpenSSL for the first time. It is based on a reproduction by Filip Skokan in #66903. Refs: #66903 Refs: #66935 Signed-off-by: Christian Aurich Zanettini Martins <christian.aurichzm@gmail.com> PR-URL: #66966 Reviewed-By: Filip Skokan <panva.ip@gmail.com>
OpenSSL 3 builds its table of algorithm names on first use. When 43d3fe9 moved that first use after other threads had started, an RSA-PSS key decoded while the table was being built could fail with `ERR_OSSL_EVP_DECODE_ERROR`, and every later RSA-PSS decode in the process failed too. #66903 reverted that change. The new test fails if it comes back: Workers decode an RSA-PSS certificate while the root certificate loader or a thread pool crypto job uses OpenSSL for the first time. It is based on a reproduction by Filip Skokan in #66903. Refs: #66903 Refs: #66935 Signed-off-by: Christian Aurich Zanettini Martins <christian.aurichzm@gmail.com> PR-URL: #66966 Reviewed-By: Filip Skokan <panva.ip@gmail.com>
OpenSSL 3 builds its table of algorithm names on first use. When 43d3fe9 moved that first use after other threads had started, an RSA-PSS key decoded while the table was being built could fail with `ERR_OSSL_EVP_DECODE_ERROR`, and every later RSA-PSS decode in the process failed too. #66903 reverted that change. The new test fails if it comes back: Workers decode an RSA-PSS certificate while the root certificate loader or a thread pool crypto job uses OpenSSL for the first time. It is based on a reproduction by Filip Skokan in #66903. Refs: #66903 Refs: #66935 Signed-off-by: Christian Aurich Zanettini Martins <christian.aurichzm@gmail.com> PR-URL: #66966 Reviewed-By: Filip Skokan <panva.ip@gmail.com>
This reverts commits 43d3fe9 and 6ecfd65.
I keep finding fallout from this change. It already needed special handling for AIX and OpenSSL provider initialization, and changed behavior with unavailable DRBGs and
--secure-heap. Since landing, I've had to marktest-worker-init-failureflaky on SmartOS because of descriptor-dependent entropy failures. Now I'm seeing intermittenttest-crypto-x509failures across GHA and Jenkins.The X509 failures come from removing eager OpenSSL initialization: background CA loading can race with key decoding while OpenSSL is still populating its algorithm names. Reverting restores initialization before that concurrency starts.
The startup improvement isn't in my opinion worth continuing to chase these failures.