openssl s_client -cert: Proving a client certificate was sent to the server

Question

Background

I am stuck in a finger-pointing match with a service provider with an API protected by SSL server and client certificates.

• I have generated a CSR, obtained a certificate from a public CA (GoDaddy in this case) and provided the certificate and CA chain to the service provider.
• They have supposedly loaded the the CA and my client certificate into their gateway.
• I am working with the most basic level tests using openssl s_client -connect ... -cert ... -key ...
• The provider tells me that their logs suggest my requests do not include a client SSL certificate at all.
• Strangely, the proper CA issuer for my certificate does appear in list of "Acceptable client certificate CA names" provided during the SSL handshake.
• For reference, a self-signed certificate I created and provided to them for testing does in fact work properly.

A sample (failed) request

[shell ~]$ openssl s_client -connect host:443 -cert cert_and_key.pem -key cert_and_key.pem -state -quiet
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=2 **SNIP**
verify return:1
depth=1 **SNIP**
verify return:1
depth=0 **SNIP**
verify return:1
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server certificate request A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client certificate A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write certificate verify A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL3 alert read:fatal:unknown CA
SSL_connect:failed in SSLv3 read finished A
140011313276744:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:s3_pkt.c:1197:SSL alert number 48
140011313276744:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:

My reading of the SSL3 alert read:fatal:unknown CA error is that the server does not recognize the issuer of the certificate I am (in fact) providing. However, the provider "assures" me that the CA certificates are loaded properly and I have been unable to convince them otherwise.

Question

So, putting other (extensive) troubleshooting steps aside, what I'd really like to know is:

Is there some output available from openssl s_client that conclusively shows that a client certificate wasn't just requested by the server, but in fact was transmitted to the server during the SSL handshake?

I have experimented with the -state, -msg, -debug and -trace options, but don't have the background necessary to interpret the output.

EJP's answer suggests that the sample output I provided is proof enough with the write client certificate A, but this output appears regardless of whether the -cert options was used on the command line or not, so it's not indicative that a certificate was sent.

Answer 1

In order to verify a client certificate is being sent to the server, you need to analyze the output from the combination of the -state and -debug flags.

First as a baseline, try running

$ openssl s_client -connect host:443 -state -debug

You'll get a ton of output, but the lines we are interested in look like this:

SSL_connect:SSLv3 read server done A
write to 0x211efb0 [0x21ced50] (12 bytes => 12 (0xC))
0000 - 16 03 01 00 07 0b 00 00-03                        .........
000c - <SPACES/NULS>
SSL_connect:SSLv3 write client certificate A

What's happening here:

• The -state flag is responsible for displaying the end of the previous section:

  SSL_connect:SSLv3 read server done A  
  

  This is only important for helping you find your place in the output.

• Then the -debug flag is showing the raw bytes being sent in the next step:

  write to 0x211efb0 [0x21ced50] (12 bytes => 12 (0xC))
  0000 - 16 03 01 00 07 0b 00 00-03                        .........
  000c - <SPACES/NULS>
  

• Finally, the -state flag is once again reporting the result of the step that -debug just echoed:

  SSL_connect:SSLv3 write client certificate A
  

So in other words: s_client finished reading data sent from the server, and sent 12 bytes to the server as (what I assume is) a "no client certificate" message.

---

If you repeat the test, but this time include the -cert and -key flags like this:

$ openssl s_client -connect host:443 \
   -cert cert_and_key.pem \
   -key cert_and_key.pem  \
   -state -debug

your output between the "read server done" line and the "write client certificate" line will be much longer, representing the binary form of your client certificate:

SSL_connect:SSLv3 read server done A
write to 0x7bd970 [0x86d890] (1576 bytes => 1576 (0x628))
0000 - 16 03 01 06 23 0b 00 06-1f 00 06 1c 00 06 19 31   ....#..........1
(*SNIP*)
0620 - 95 ca 5e f4 2f 6c 43 11-                          ..^%/lC.
SSL_connect:SSLv3 write client certificate A

The 1576 bytes is an excellent indication on its own that the cert was transmitted, but on top of that, the right-hand column will show parts of the certificate that are human-readable: You should be able to recognize the CN and issuer strings of your cert in there.