MariaDB over SSL not working, "certificate verify failed"
Using this guide I'm attempting to set up MariaDB (mysql) for using SSL between dbserver and appclient.
I created the server and client certificates on the server, per the guide. I then copied the three necessary client files to appclient and set ownership and permissions:
[root@appclient mysql]# ll /etc/pki/tls/certs/
drwxr-xr-x. 2 mysql mysql 88 Feb 9 13:31 mysql
[root@appclient mysql]# ll /etc/pki/tls/certs/mysql/
-rw-------. 1 mysql mysql 1372 Feb 9 13:31 ca-cert.pem
-rw-------. 1 mysql mysql 1230 Feb 9 14:16 client-cert.pem
-rw-------. 1 mysql mysql 1705 Feb 9 14:16 client-key.pem
Here's the full my.cnf on appclient:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
[client]
ssl-ca=/etc/pki/tls/certs/mysql/ca-cert.pem
ssl-cert=/etc/pki/tls/certs/mysql/client-cert.pem
ssl-key=/etc/pki/tls/certs/mysql/client-key.pem
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
!includedir /etc/my.cnf.d
Next, I tested that port 3306 is open on dbserver:
[root@appclient mysql]# telnet dbserver 3306
Connected to dbserver.
Escape character is '^]'.
R
5.5.52-MariaDB
Next I checked MariaDB (mysql) ssl variables on dbserver:
MariaDB [(none)]> show variables like '%ssl%';
+---------------+------------------------------------------+
| Variable_name | Value |
+---------------+------------------------------------------+
| have_openssl | YES |
| have_ssl | YES |
| ssl_ca | /etc/pki/tls/certs/mysql/ca-cert.pem |
| ssl_capath | |
| ssl_cert | /etc/pki/tls/certs/mysql/server-cert.pem |
| ssl_cipher | |
| ssl_key | /etc/pki/tls/certs/mysql/server-key.pem |
+---------------+------------------------------------------+
Next I checked MariaDB (mysql) ssl variables on appclient:
MariaDB [(none)]> show variables LIKE '%ssl%';
+---------------+----------+
| Variable_name | Value |
+---------------+----------+
| have_openssl | DISABLED |
| have_ssl | DISABLED |
| ssl_ca | |
| ssl_capath | |
| ssl_cert | |
| ssl_cipher | |
| ssl_key | |
+---------------+----------+
7 rows in set (0.00 sec)
That looks like the start/source of the problem.
If I try to connect to dbserver from appclient anyway:
[root@appclient mysql]# mysql -h dbserver -u ssluser -p
Enter password:
ERROR 2026 (HY000): SSL connection error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
No bueno.
Checking appclient's certs with openssl...
[root@appclient mysql]# cd /etc/pki/tls/certs/mysql/
[root@appclient mysql]# openssl verify -CAfile ca-cert.pem server-cert.pem client-cert.pem
Error opening certificate file server-cert.pem
139864320337824:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('server-cert.pem','r')
139864320337824:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:
unable to load certificate
client-cert.pem: OK
For kicks, I ran the same openssl test on dbserver:
[root@dbserver mysql]# openssl verify -CAfile ca-cert.pem server-cert.pem client-cert.pem
server-cert.pem: C = XX, ST = XX, L = CityName, O = MyOrganization, OU = MyGroup, CN = dbserver
error 18 at 0 depth lookup:self signed certificate
OK
client-cert.pem: OK
The tutorial only mentions copying ca-cert.pem, client-cert.pem and client-key.pem to the client, yet the failure above points to a missing server-cert.pem on the client.
Do I need to create the server-*.pem files on the client also? If so, where do these go in the /etc/my.cnf file?
Answer
The missing ingredient from several of the MySQL/MariaDB SSL setup guides is making sure the ssl-ca certificate file contains both server and client ca's.
Here's a step by step guide which worked for me:
This answer assumes two servers:
- dbserver (where our database lives)
- appclient (where our applications live)
FWIW, both servers are SELinux enforcing.
First, log on to dbserver
Create a temporary directory for creating the certificates.
mkdir /root/certs/mysql/ && cd /root/certs/mysql/
Create the server certificates
openssl genrsa 2048 > ca-key.pem
openssl req -sha1 -new -x509 -nodes -days 3650 -key ca-key.pem > ca-cert.pem
openssl req -sha1 -newkey rsa:2048 -days 730 -nodes -keyout server-key.pem > server-req.pem
openssl rsa -in server-key.pem -out server-key.pem
openssl x509 -sha1 -req -in server-req.pem -days 730 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 > server-cert.pem
Move server certificates to /etc/pki/tls/certs/mysql/ Directory path assumes CentOS or RHEL (adjust as needed for other distros):
mkdir /etc/pki/tls/certs/mysql/
cp /root/certs/mysql/* /etc/pki/tls/certs/mysql/
Be sure to set permissions on the folder and files. mysql needs full ownership and access.
chown -R mysql:mysql /etc/pki/tls/certs/mysql
Now configure MySQL/MariaDB
# vi /etc/my.cnf
# i
[mysqld]
bind-address=*
ssl-ca=/etc/pki/tls/certs/ca-cert.pem
ssl-cert=/etc/pki/tls/certs/server-cert.pem
ssl-key=/etc/pki/tls/certs/server-key.pem
# :wq
Then
systemctl restart mariadb
Don't forget to open your firewall to allow connections from appclient (using IP 1.2.3.4)
firewall-cmd --zone=drop --permanent --add-rich-rule 'rule family="ipv4" source address="1.2.3.4" service name="mysql" accept'
# I force everything to the drop zone. Season the above command to taste.
Now restart firewalld
service firewalld restart
Next, log in to dbserver's mysql server:
mysql -uroot -p
Issue the following to create a user for the client. note REQUIRE SSL in GRANT statement.
GRANT ALL PRIVILEGES ON *.* TO ‘iamsecure’@’appclient’ IDENTIFIED BY ‘dingdingding’ REQUIRE SSL;
FLUSH PRIVILEGES;
# quit mysql
You should still be in /root/certs/mysql from the first step. If not, cd back to it for one of the commands below.
Create the client certificates
openssl req -sha1 -newkey rsa:2048 -days 730 -nodes -keyout client-key.pem > client-req.pem
openssl rsa -in client-key.pem -out client-key.pem
openssl x509 -sha1 -req -in client-req.pem -days 730 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 > client-cert.pem
Note: I used the same common name for both server and client certificates. YMMV.
Be sure you're still /root/certs/mysql/ for this next command
Combine server and client CA certificate into a single file:
cat server-cert.pem client-cert.pem > ca.pem
Make sure you see two certificates:
cat ca.pem
END OF SERVER SIDE WORK FOR NOW.
Open another terminal and
ssh appclient
As before, create a permanent home for the client certificates
mkdir /etc/pki/tls/certs/mysql/
Now, place the client certificates (created on dbserver) on appclient. You can either scp them over, or just copy and paste the files one by one.
scp dbserver
# copy files from dbserver to appclient
# exit scp
Again, be sure to set permissions on the folder and files. mysql needs full ownership and access.
chown -R mysql:mysql /etc/pki/tls/certs/mysql
You should have three files, each owned by user mysql:
/etc/pki/tls/certs/mysql/ca.pem
/etc/pki/tls/certs/mysql/client-cert.pem
/etc/pki/tls/certs/mysql/client-key.pem
Now edit appclient's MariaDB/MySQL config in the [client] section.
vi /etc/my.cnf
# i
[client]
ssl-ca=/etc/pki/tls/certs/mysql/ca.pem
ssl-cert=/etc/pki/tls/certs/mysql/client-cert.pem
ssl-key=/etc/pki/tls/certs/mysql/client-key.pem
# :wq
Restart appclient's mariadb service:
systemctl restart mariadb
still on the client here
This should return: ssl TRUE
mysql --ssl --help
Now, log in to appclient's mysql instance
mysql -uroot -p
Should see YES to both variables below
show variables LIKE '%ssl';
have_openssl YES
have_ssl YES
Initially I saw
have_openssl NO
A quick look into mariadb.log revealed:
SSL error: Unable to get certificate from '/etc/pki/tls/certs/mysql/client-cert.pem'
The problem was that root owned client-cert.pem and the containing folder. The solution was to set ownership of /etc/pki/tls/certs/mysql/ to mysql.
chown -R mysql:mysql /etc/pki/tls/certs/mysql
Restart mariadb if needed from the step immediately above
NOW WE ARE READY TO TEST THE SECURE CONNECTION
We're still on appclient here
Attempt to connect to dbserver's mysql instance using the account created above.
mysql -h dbserver -u iamsecure -p
# enter password dingdingding (hopefully you changed that to something else)
With a little luck you should be logged in without error.
To confirm you are connected with SSL enabled, issue the following command from the MariaDB/MySQL prompt:
\s
That's a backslash s, aka status
That will show the status of your connection, which should look something like this:
Connection id: 4
Current database:
Current user: iamsecure@appclient
SSL: Cipher in use is DHE-RSA-AES256-GCM-SHA384
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server: MariaDB
Server version: 5.X.X-MariaDB MariaDB Server
Protocol version: 10
Connection: dbserver via TCP/IP
Server characterset: latin1
Db characterset: latin1
Client characterset: utf8
Conn. characterset: utf8
TCP port: 3306
Uptime: 42 min 13 sec
If you get permission denied errors on your connection attempt, check your GRANT statement above to make sure there aren't any stray characters or ' marks.
If you have SSL errors, go back through this guide to make sure the steps are orderly.
This worked on RHEL7 and will likely work on CentOS7, too. Cannot confirm whether these exact steps will work elsewhere.
Hope this saves someone else a little time and aggravation.