I am running SCEP client to enroll certificates on NDES server. If OpenSSL is not in FIPS mode - everything works fine.

In FIPS mode i get the following error:

pkcs7_unwrap():pkcs7.c:708] error decrypting inner PKCS#7
139968442623728:error:060A60A3:digital envelope routines:FIPS_CIPHERINIT:disabled for fips:fips_enc.c:142:
139968442623728:error:21072077:PKCS7 routines:PKCS7_decrypt:decrypt error:pk7_smime.c:557:

That's because NDES server uses DES algorithm to encrypt returned PKCS#7 packet. I used the following debug code:

/* Copy enveloped data from PKCS#7 */
    bytes = BIO_read(pkcs7bio, buffer, sizeof(buffer));
    BIO_write(outbio, buffer, bytes);

    p7enc = d2i_PKCS7_bio(outbio, NULL);

/* Get encryption  PKCS#7 algorithm */

    enc_alg=p7enc->d.enveloped->enc_data->algorithm;
    evp_cipher=EVP_get_cipherbyobj(enc_alg->algorithm);
    printf("evp_cipher->nid = %d\n", evp_cipher->nid);

The last string always prints:

evp_cipher->nid = 31

defined in openssl-1.0.1c/include/openssl/objects.h

#define SN_des_cbc "DES-CBC"
#define LN_des_cbc "des-cbc"
#define NID_des_cbc 31

I use 3DES algorithm for PKCS7 requests encryption in my code (pscep.enc_alg = (EVP_CIPHER *)EVP_des_ede3_cbc()) and NDES server accepts these requests, but it always returns answer encrypted with DES.

Can I configure Wndows NDES server to use Triple DES (3DES) algorithm for PKCS#7 answer encryption?