Scenario: calling a client web service over SSL (https) with mutual SSL authentication. Different service endpoint URLs and certs (both keystore and truststore) for test vs. production environments. Both test and production environments run tomcat / JBoss clustered. Production environment has load balancing / BigIP, runs Blade and non-Blade machines.

Truststore is set (using -Djavax.net.ssl.trustStore=value) at startup. Keystore is set using System.setProperty("javax.net.ssl.keyStore", "value") in Java code. Web service call made using Axis2. All works fine in test environment, but when we moved to production environment (6 servers), it appears certs are not being forwarded for the handshake. Here's what we've done:

• in test environment, handshake using test versions of certs has been working all along, with no ssl debugging enabled
• confirmed in test environment that handshake with client production endpoint succeeds (production certs, both ours and theirs, are fine) -- this was done using -Djavax.net.debug=handshake,ssl
• confirmed that the error condition occurs on all 6 production servers
• took one server out of the cluster, turned on ssl debugging for just that one (with a restart), hit it directly, handshake works!
• switched to a different server without the debugging turned on, handshake error condition occurs
• turned debugging on on that second server (with a restart), hit it directly, handshake works!

From the evidence, it seems like somehow the debugging being enabled causes the certificates to be properly retrieved/conveyed, although that makes no sense! I wonder whether somehow the enabled debugging makes the system pay attention to the System.setProperty call, and ignore it otherwise. However, in local and test environments, handshake worked without debugging enabled.

Do I maybe need to be setting keystore on server startup like I'm setting truststore? Have been avoiding that because the keystore will differ for each of our test environments (16 of them).