1. Overview
A Secured Socket Layer (SSL) is a cryptographic protocol that provides security in communication over the network. In this tutorial, we’ll discuss various scenarios that can result in an SSL handshake failure and how to do it.
Note that our Introduction to SSL using JSSE covers the basics of SSL in more detail.
2. Terminology
It’s important to note that, due to security vulnerabilities, Transport Layer Security (TLS) supersedes SSL as a standard. Most programming languages, including Java, have libraries to support both SSL and TLS.
Since the inception of SSL, many products and languages like OpenSSL and Java had references to SSL, which they kept even after TLS took over. For this reason, in the remainder of this tutorial, we will use the term SSL to refer generally to cryptographic protocols.
3. Setup
To demonstrate the workings, we’ll create server-client applications using the Java Socket API and simulate a network connection.
3.1. Creating a Client and a Server
In Java, we can use sockets to establish a communication channel between a server and a client over the network. Sockets are a part of the Java Secure Socket Extension (JSSE) in Java.
Let’s begin by defining a simple server:
int port = 8443;
ServerSocketFactory factory = SSLServerSocketFactory.getDefault();
try (ServerSocket listener = factory.createServerSocket(port)) {
SSLServerSocket sslListener = (SSLServerSocket) listener;
sslListener.setNeedClientAuth(true);
sslListener.setEnabledCipherSuites(
new String[] { "TLS_DHE_DSS_WITH_AES_256_CBC_SHA256" });
sslListener.setEnabledProtocols(
new String[] { "TLSv1.2" });
while (true) {
try (Socket socket = sslListener.accept()) {
PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
out.println("Hello World!");
}
}
}
The server defined above returns the message “Hello World!” to a connected client.
Next, let’s define a client that will connect to our SimpleServer and read messages from the server:
String host = "localhost";
int port = 8443;
SocketFactory factory = SSLSocketFactory.getDefault();
try (Socket connection = factory.createSocket(host, port)) {
((SSLSocket) connection).setEnabledCipherSuites(
new String[] { "TLS_DHE_DSS_WITH_AES_256_CBC_SHA256" });
((SSLSocket) connection).setEnabledProtocols(
new String[] { "TLSv1.2" });
SSLParameters sslParams = new SSLParameters();
sslParams.setEndpointIdentificationAlgorithm("HTTPS");
((SSLSocket) connection).setSSLParameters(sslParams);
BufferedReader input = new BufferedReader(
new InputStreamReader(connection.getInputStream()));
return input.readLine();
}
Our client prints the message returned by the server.
3.2. Creating Certificates in Java
SSL provides secrecy, integrity, and authenticity in network communications. Certificates play an essential role as far as establishing authenticity.
In general, we purchase a certificate from a Certificate Authority. But we’ll use a self-signed certificate for our examples.
To achieve this, we can use keytool, which ships with the JDK:
$ keytool -genkey -keypass password \
-storepass password \
-keystore serverkeystore.jks
The above command starts an interactive shell to gather information for the certificate, like Common Name (CN) and Distinguished Name (DN). When we provide all relevant details, it generates the file serverkeystore.jks, which contains the private key of the server and its public certificate.
Note that serverkeystore.jks uses the Java Key Store (JKS) format, which is proprietary to Java. These days, keytool will remind us that we ought to consider using PKCS#12, which it also supports.
Further, we can use keytool to extract the public certificate from the generated keystore file:
$ keytool -export -storepass password \
-file server.cer \
-keystore serverkeystore.jks
In short, the command above exports the public certificate from the keystore as a file server.cer. Now, let’s add the certificate to the client’s truststore:
$ keytool -import -v -trustcacerts \
-file server.cer \
-keypass password \
-storepass password \
-keystore clienttruststore.jks
We have now generated a keystore for the server and a corresponding truststore for the client. We will go over the use of these generated files when we discuss possible handshake failures.
4. SSL Handshake
SSL handshakes are a mechanism by which a client and server establish the trust and logistics required to secure their connection over the network. Understanding the details of this orchestrated procedure is crucial to debug failures.
Typical steps in an SSL handshake are:
- The client provides a list of possible SSL versions and cipher suites to use
- The server agrees on a particular SSL version and cipher suite, responding with its certificate
- The client extracts the public key from the certificate and responds with an encrypted “pre-master key”
- The server decrypts the “pre-master key” using its private key
- Client and server compute a “shared secret” using the exchanged “pre-master key”
- Client and server exchange messages confirming the successful encryption and decryption using the “shared secret”
While most of the steps are the same for any SSL handshake, there is a subtle difference between one-way and two-way handshakes. Let’s quickly review these differences.
4.1. The Handshake in One-way SSL
If we refer to the steps mentioned above, step two mentions the certificate exchange. One-way SSL requires that a client can trust the server through its public certificate. This leaves the server to trust all clients that request a connection. There is no way for a server to request and validate the public certificate from clients, which can pose a security risk.
4.2. The Handshake in Two-way SSL
With one-way SSL, the server must trust all clients. But, two-way SSL adds the ability for the server to be able to establish trusted clients as well. To establish a successful connection using a two-way handshake, both the client and server must present and accept each other’s public certificates.
5. Handshake Failure Scenarios
Having done that quick review, we can look at failure scenarios with greater clarity.
An SSL handshake, in one-way or two-way communication, can fail for multiple reasons. We will go through each of these reasons, simulate the failure, and understand how we can avoid such scenarios. In each of these scenarios, we will use the SimpleClient and SimpleServer from our previous example.
Most times, the exception thrown in case of failure will be a generic one. Therefore, to debug the ssl handshake, we must set the javax.net.debug property to ssl:handshake to show us more granular details about the handshake:
System.setProperty("javax.net.debug", "ssl:handshake");
5.1. Missing Server Certificate
Let’s try to run the SimpleServer and connect it through the SimpleClient. Straightaway, the SimpleClient throws an exception instead of the “HelloWorld!” message:
Exception in thread "main" javax.net.ssl.SSLHandshakeException:
Received fatal alert: handshake_failure
Now, this indicates something went wrong. The SSLHandshakeException above, in an abstract manner, states that the client, when connecting to the server, did not receive any certificate.
To address this issue, we will use the keystore we generated earlier by passing them as system properties to the server:
-Djavax.net.ssl.keyStore=serverkeystore.jks -Djavax.net.ssl.keyStorePassword=password
It’s important to note that the system property for the keystore file path should either be an absolute path or the keystore file should be placed in the same directory from where the Java command is invoked to start the server. Java system property for keystore does not support relative paths.
Does this help us get the output we are expecting? Let’s find out in the following sub-section.
5.2. Untrusted Server Certificate
As we rerun the SimpleServer and the SimpleClient with the changes in the previous sub-section, what do we get as output:
Exception in thread "main" javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException:
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target
Well, it did not work exactly as we expected, but it looks like it has failed for a different reason.
This particular failure is caused by the fact that our server is using a self-signed certificate, which is not signed by a Certificate Authority (CA).
Really, any time the certificate is signed by something other than what is in the default truststore, we’ll see this error. The default truststore in JDK typically ships with information about common CAs in use.
To solve this issue, we will have to force SimpleClient to trust the certificate presented by SimpleServer. Let’s use the truststore we generated earlier by passing them as system properties to the client:
-Djavax.net.ssl.trustStore=clienttruststore.jks -Djavax.net.ssl.trustStorePassword=password
Please note that this is not an ideal solution. In an ideal scenario, we should not use a self-signed certificate but a certificate that has been certified by a Certificate Authority (CA), which clients can trust by default.
Let’s go to the following sub-section to find out if we get our expected output now.
5.3. Missing Client Certificate
Let’s try one more time running the SimpleServer and the SimpleClient, having applied the changes from previous sub-sections:
Exception in thread "main" java.net.SocketException:
Software caused connection abort: recv failed
Again, not something we expected. The SocketException here tells us that the server could not trust the client. This is because we have set up a two-way SSL. In our SimpleServer we have:
((SSLServerSocket) listener).setNeedClientAuth(true);
The above code indicates an SSLServerSocket is required for client authentication through their public certificate.
We can create a keystore for the client and a corresponding truststore for the server in a way similar to the one that we used when creating the previous keystore and truststore.
We will restart the server and pass it the following system properties:
-Djavax.net.ssl.keyStore=serverkeystore.jks \
-Djavax.net.ssl.keyStorePassword=password \
-Djavax.net.ssl.trustStore=clienttruststore.jks \
-Djavax.net.ssl.trustStorePassword=password
Then, we will restart the client by passing these system properties:
-Djavax.net.ssl.keyStore=serverkeystore.jks \
-Djavax.net.ssl.keyStorePassword=password \
-Djavax.net.ssl.trustStore=clienttruststore.jks \
-Djavax.net.ssl.trustStorePassword=password
Finally, we have the output we desired:
Hello World!
5.4. Incorrect Certificates
Apart from the above errors, a handshake can fail due to a variety of reasons related to how we have created the certificates. One recurring error relates to an incorrect CN. Let’s explore the details of the server keystore we created previously:
keytool -v -list -keystore serverkeystore.jks
When we run the above command, we can see the details of the keystore, specifically the owner:
...
Owner: CN=localhost, OU=technology, O=baeldung, L=city, ST=state, C=xx
...
The CN of the owner of this certificate is set to localhost. The CN of the owner must exactly match the host of the server. If there is any mismatch, it will result in an SSLHandshakeException.
Let’s try to regenerate the server certificate with CN as anything other than localhost. When we use the regenerated certificate now to run the SimpleServer and SimpleClient, it promptly fails:
Exception in thread "main" javax.net.ssl.SSLHandshakeException:
java.security.cert.CertificateException:
No name matching localhost found
The exception trace above clearly indicates that the client was expecting a certificate bearing the name localhost, which it did not find.
Please note that JSSE does not mandate hostname verification by default. We have enabled hostname verification in the SimpleClient through explicit use of HTTPS:
SSLParameters sslParams = new SSLParameters();
sslParams.setEndpointIdentificationAlgorithm("HTTPS");
((SSLSocket) connection).setSSLParameters(sslParams);
Hostname verification is a common cause of failure and, in general, should consistently be enforced for better security. For details on hostname verification and its importance in security with TLS, please refer to this article.
5.5. Incompatible SSL Version
Currently, there are various cryptographic protocols, including different versions of SSL and TLS, in operation.
As mentioned earlier, SSL, in general, has been superseded by TLS for its cryptographic strength. The cryptographic protocol and version are an additional element that a client and a server must agree on during a handshake.
For example, if the server uses a cryptographic protocol of SSL3 and the client uses TLS1.3, they cannot agree on a cryptographic protocol, and an SSLHandshakeException will be generated.
In our SimpleClient, let’s change the protocol to something that is not compatible with the protocol set for the server:
((SSLSocket) connection).setEnabledProtocols(new String[] { "TLSv1.1" });
When we run our client, we will get an SSLHandshakeException:
Exception in thread "main" javax.net.ssl.SSLHandshakeException:
No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
The exception trace in such cases is abstract and does not tell us the exact problem. To resolve these types of problems, it is necessary to verify that both the client and server are using either the same or compatible cryptographic protocols.
5.6. Incompatible Cipher Suite
The client and server must also agree on the cipher suite they will use to encrypt messages.
During a handshake, the client will present a list of possible ciphers to use, and the server will respond with a selected cipher from the list. The server will generate an SSLHandshakeException if it cannot select a suitable cipher.
In our SimpleClient, let’s change the cipher suite to something that is not compatible with the cipher suite used by our server:
((SSLSocket) connection).setEnabledCipherSuites(
new String[] { "TLS_RSA_WITH_AES_128_GCM_SHA256" });
When we restart our client, we will get an SSLHandshakeException:
Exception in thread "main" javax.net.ssl.SSLHandshakeException:
Received fatal alert: handshake_failure
Again, the exception trace is quite abstract and does not tell us the exact problem. The resolution to such an error is to verify the enabled cipher suites used by both the client and server and ensure that there is at least one common suite available.
Usually, clients and servers are configured to use a wide variety of cipher suites, so this error is less likely to happen. If we encounter this error, it is typically because the server has been configured to use a very selective cipher. A server may choose to enforce a selective set of ciphers for security reasons.
6. Conclusion
In this tutorial, we learned about setting up SSL using Java sockets. Then, we discussed SSL handshakes with one-way and two-way SSL. Finally, we went through a list of possible reasons that SSL handshakes may fail and discussed the solutions.
As always, the code for the examples is available over on GitHub.
