OpenSSL is the most widely deployed crypto library in the world. Most ICT companies use OpenSSL in one one way or another. Thus the FIPS validation of OpenSSL has been tremendously useful to the industry as it provides a handy shortcut to FIPS compliance. However, the openSSL FIPS validation is not without issues. The original openSSL validation was revoked due to a non-compliant RNG. More recently there have been issues with the way openSSL runs power-on self-tests. Additionally, openSSL FIPS validation has not kept pace with newer algorithms such as FIPS 186-4 RSA key generation, CVL certifications, etc.

Today it seems a major FIPS related flaw has been uncovered with openSSL. On the openssl-users group, there was an exchange between Randy Steck and Dr. Stephen Henson on how RSA key generation function is called by the API exported by libcrypto.

Here are the deets:

OpenSSL FOM exports two RSA key generation related APIs:

  • FIPS_rsa_x931_generate_key_ex(): This is the function that implements FIPS 186-2 compliant RSA key generation
  • FIPS_rsa_generate_key_ex(): This is a wrapper function that by default calls rsa_builtin_keygen() which does not conform to FIPS 186-2 (or the newer FIPS 186-3/4). Additionally the FIPS_rsa_generate_key_ex() function takes in a data structure where you can specify a custom key generation method (e.g. to call RSA key generation that might be implemented in a HW crypto accelerator).

As you can see above to call the FIPS compliant RSA key generation function you have to specifically call FIPS_rsa_x931_generate_key_ex() or ensure that a custom method (that is compliant with FIPS) is defined in the input structure to FIPS_rsa_generate_key_ex(). The bug here is that in FIPS mode, FIPS_rsa_generate_key_ex() should have defaulted to FIPS_rsa_x931_generate_key_ex().

However the bigger issue is how the larger openSSL library ( calls down to openSSL FOM for RSA key generation. In our experience, rarely (if at all) calls are made directly to the FOM. In most cases, openSSL was already in the code infrastructure and when a product vendor decides to pursue FIPS, they just drop replace the crypto core with FIPS FOM. This makes sense as it significantly reduces code re-work and the original openSSL API calls can be retained unchanged. OpenSSL exports RSA_generate_key_ex() API for RSA key generation. The call path for this API is as follows:

  • non-FIPS mode: RSA_generate_key_ex() -> rsa_builtin_keygen()
  • FIPS mode: RSA_generate_key_ex() -> FIPS_rsa_generate_key_ex() -> FIPS_rsa_builtin_keygen()

As you can see if the correct FIPS compliant RSA key generation method is not defined in the call to RSA_generate_key_ex() the call path goes to the non-FIPS compliant RSA key generation in FIPS mode.

Moreover, if you use the RSA key generation command (openssl genrsa) the non-FIPS approved RSA key generation method is used even in FIPS mode of operation.

So the obvious question is what should you as the developer do. There are three options:

  1. Modify FIPS_rsa_generate_key_ex() to default to FIPS_rsa_x931_generate_key_ex(). This is the cleaner option however it will likely invalidated openSSL FOM’s FIPS certification
  2. Ensure that your calls to RSA_generate_key_ex() pass in the correct FIPS compliant RSA key generation method in the *rsa data struct. This keeps the FOM FIPS validation intact however it does require making changes to all your RSA key generation calls.
  3. Call FIPS_rsa_x931_generate_key_ex() directly whenever RSA keys need to be generated. This could potentially mean significant re-write of existing code and probably the least palatable option.

One final note to keep in mind, openSSL’s RSA key generation method (implemented in FIPS_rsa_x931_generate_key_ex()) is compliant with with the older ANSI X9.31 key generation which has been deprecated by FIPS 186-4. So, technically, the key generation cannot be validated (for new validations) regardless of whether the correct call sequence is made or not. In order to be compliant with latest FIPS 140-2 requirements, vendors will have to fix this flaw (i.e. call the correct RSA key generation function) and modify the key generation implementation to comply with FIPS 86-4.

Please feel free to contact me if you have questions about this.

Update: The OpenSSL code (note that this is not the FOM code which is the FIPS crypto boundary) has been updated to ensure that the correct function is called. See code diffs here.