"unknown ca" with self-generated CA, certificates and client/server

Question

I'm writing a custom client & server that I want to communicate securely over the public Internet, therefore I want to use OpenSSL and have both ends do peer verification to ensure that my client isn't mis-directed by a MITM, and likewise that an unauthorized client isn't able to connect to the server.

This is the error received from the server during the SSL_connect / SSL_accept phase:

15620:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:ssl\record\rec_layer_s3.c:1528:SSL alert number 48

I'm running under Windows 10, using OpenSSL 1.1.1. I'm using the following batch file to create them. I enter the ca private key passphrase by hand for obvious reasons.

openssl genrsa -out -des3 ca.key.pem 2048
openssl genrsa -out server.key.pem 2048
openssl genrsa -out client.key.pem 2048

openssl req -x509 -new -nodes -key ca.key.pem -sha256 -days 365 -out ca.cert.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar

openssl req -new -sha256 -key server.key.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar -out server.csr
openssl x509 -req -in server.csr -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out server.cert.pem -days 365 -sha256

openssl req -new -sha256 -key client.key.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=Foobar -out client.csr
openssl x509 -req -in client.csr -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -out client.cert.pem -days 365 -sha256

The intent here is to create a self-signed CA, and then have that directly sign both the client and server keys.

ca.key.pem will be stored in a secure place: on an encrypted veracrypt volume.

Both client and server use the following call to enable peer verification:

    SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT, nullptr);

I'm fairly certain this is a certificate issue because the errors go away if I remove that line.

Answer 1

Answering this myself so that it can help anyone else that might arrive here looking for solutions to this problem. The answer was found in another SO question, but is worth repeating here: The Common Name for the CA cannot be the same as the Common Name for the client and server certificates.

So changing the fourth line of the batch file to this:

openssl req -x509 -new -nodes -key ca.key.pem -sha256 -days 365 -out ca.cert.pem -subj /C=US/ST=CA/L=Somewhere/O=Someone/CN=FoobarCA

fixed the problem.