Fix UAF in OpenSSL.fixed_length_secure_compare

[apocalypse9949]

This PR fixes a Use-After-Free (UAF) vuln in OpenSSL.fixed_length_secure_compare.

Description

The vuln existed because StringValuePtr(str) and RSTRING_PTR(str) macro calls were performed on the first argument before StringValue() was called on the second argument.

Since StringValue() implicitly calls #to_str on arguments that are not already Strings, an attacker could supply an object whose #to_str method modifies the other argument's content. If the first string argument is sufficiently mutated, its buffer will be reallocated. This leaves the initial p1 pointer pointing to freed memory, leading to a UAF when CRYPTO_memcmp is called.

Fix

This patch resolves the UAF by calling StringValue() on both string arguments first, ensuring they are fully evaluated and any #to_str side effects occur before taking the C pointers to their underlying string buffers.

A regression test case has also been included in test/openssl/test_ossl.rb to cover this exact scenario.

[github-actions]

The following files are maintained in the following upstream repositories:

• https://github.com/ruby/openssl
  • ext/openssl/ossl.c
  • test/openssl/test_ossl.rb

Please file a pull request to the above instead. Thank you!

[rhenium]

Moved to ruby/openssl: ruby/openssl#1016