The BN_mod_sqrt()
function, which computes a modular square root, contains
a bug that can cause it to loop forever for non-prime moduli.
Internally this function is used when parsing certificates that contain elliptic curve public keys in compressed form or explicit elliptic curve parameters with a base point encoded in compressed form.
It is possible to trigger the infinite loop by crafting a certificate that has invalid explicit curve parameters.
Since certificate parsing happens prior to verification of the certificate signature, any process that parses an externally supplied certificate may thus be subject to a denial of service attack. The infinite loop can also be reached when parsing crafted private keys as they can contain explicit elliptic curve parameters.
Thus vulnerable situations include:
Also any other applications that use the BN_mod_sqrt()
where the attacker
can control the parameter values are vulnerable to this DoS issue.