Add public CRYPTO_cts128_encrypt/_decrypt and <openssl/modes.h> for krb5 integration

[WillChilds-Klein]

Summary

AWS-LC currently has no public AES Ciphertext Stealing (CTS) API. There is an internal declaration of CRYPTO_cts128_encrypt_block in crypto/fipsmodule/modes/internal.h:411, but no implementation exists, and <openssl/modes.h> is not shipped at all.

This blocks integration with MIT krb5 (and likely other consumers) which uses CTS for the standard Kerberos enctypes.

Background

MIT krb5's lib/crypto/openssl/enc_provider/aes.c and camellia.c implement AES-CTS / Camellia-CTS for the aes128-cts-hmac-* and camellia*-cts-cmac enctypes. On the OpenSSL <3.0 path it calls:

#include <openssl/modes.h>
size = CRYPTO_cts128_encrypt(dbuf, oblock, dlen, &enck, iv_cts,
                             (cbc128_f)AES_cbc_encrypt);
size = CRYPTO_cts128_decrypt(dbuf, oblock, dlen, &deck, iv_cts,
                             (cbc128_f)AES_cbc_encrypt);

On the OpenSSL ≥3.0 path it uses EVP_CIPHER_fetch("AES-128-CBC-CTS") plus OSSL_PARAMs — provider/property infrastructure that AWS-LC does not implement and is not on a roadmap.

Since AWS-LC reports OPENSSL_VERSION_NUMBER < 0x30000000L for compatibility, krb5 takes the legacy path and tries to #include <openssl/modes.h>. AWS-LC doesn't ship that header, so the include falls through to system OpenSSL 3.x's /usr/include/openssl/modes.h, which transitively pulls OpenSSL 3.x's <openssl/types.h> and produces typedef collisions on EVP_MD, EVP_MD_CTX, ASN1_NULL, EVP_ENCODE_CTX, etc. Even if the include were resolved (e.g. by shipping a stub modes.h), CRYPTO_cts128_encrypt and CRYPTO_cts128_decrypt are still undefined at link time.

This was discovered while adding a krb5 integration test in #3308.

Asks

1. Implement CRYPTO_cts128_encrypt and CRYPTO_cts128_decrypt (CS1 / RFC 2040 RC5-CTS variant — this is what OpenSSL's legacy API does and what krb5 calls). These wrap a caller-provided cbc128_f and process arbitrary-length input ≥ one block.

2. Ship a public <openssl/modes.h> exporting:

   • cbc128_f (typedef)
   • CRYPTO_cts128_encrypt
   • CRYPTO_cts128_decrypt
   • And, for symmetry, the existing CRYPTO_cbc128_encrypt/_decrypt and CRYPTO_cfb128_* already declared internally — these are commonly imported by OpenSSL consumers.

3. Add comprehensive tests with NIST / RFC 2040 vectors covering:

   • All input lengths from 17 bytes through several full-block multiples (boundary conditions: residue == 1, residue == blocksize, residue in between)
   • Encrypt/decrypt roundtrip
   • Decrypt of a known-good ciphertext (asymmetric verification)
   • IV update behavior matches OpenSSL's

The CS3 ("always swap") variant used by krb5's OpenSSL-3.x path is not requested here; CS1 alone is what unblocks the krb5 integration on the legacy path AWS-LC is compatible with.

Acceptance criteria

• <openssl/modes.h> is installed as a public header and CRYPTO_cts128_* symbols are exported from libcrypto.
• The MIT krb5 integration test (Add CTS mode, krb5 integration test #3308) builds and passes against AWS-LC.
• New unit tests cover all input-length boundary cases and verify against fixed reference vectors.