I just installed Python3 from python.org and am having trouble installing packages with pip
. By design, there is a man-in-the-middle packet inspection appliance on the network here that inspects all packets (ssl included) by resigning all ssl connections with its own certificate. Part of the GPO pushes the custom root certificate into the Windows Keystore.
When using Java, if I need to access any external https sites, I need to manually update the cacerts in the JVM to trust the Self-Signed CA certificate.
How do I accomplish that for python? Right now, when I try to install packages using pip
, understandably, I get wonderful [SSL: CERTIFICATE_VERIFY_FAILED]
errors.
I realize I can ignore them using the --trusted-host
parameter, but I don't want to do that for every package I'm trying to install.
Is there a way to update the CA Certificate store that python uses?
pip
/ conda
After extensively documenting a similar problem with Git (How can I make git accept a self signed certificate?), here we are again behind a corporate firewall with a proxy giving us a MitM "attack" that we should trust and:
NEVER disable all SSL verification!
This creates a bad security culture. Don't be that person.
pip config set global.cert path/to/ca-bundle.crt
pip config list
conda config --set ssl_verify path/to/ca-bundle.crt
conda config --show ssl_verify
# Bonus while we are here...
git config --global http.sslVerify true
git config --global http.sslCAInfo path/to/ca-bundle.crt
But where do we get ca-bundle.crt
?
cURL publishes an extract of the Certificate Authorities bundled with Mozilla Firefox
https://curl.haxx.se/docs/caextract.html
I recommend you open up this cacert.pem
file in a text editor as we will need to add our self-signed CA to this file.
Certificates are a document complying with X.509 but they can be encoded to disk a few ways. The below article is a good read but the short version is that we are dealing with the base64 encoding which is often called PEM in the file extensions. You will see it has the format:
----BEGIN CERTIFICATE----
....
base64 encoded binary data
....
----END CERTIFICATE----
Below are a few options on how to get our self signed certificate:
echo quit | openssl s_client -showcerts -servername "curl.haxx.se" -connect curl.haxx.se:443 > cacert.pem
Thanks to this answer and the linked blog, it shows steps (on Windows) how to view the certificate and then copy to file using the base64 PEM encoding option.
Copy the contents of this exported file and paste it at the end of your cacerts.pem
file.
For consistency rename this file cacerts.pem
--> ca-bundle.crt
and place it somewhere easy like:
# Windows
%USERPROFILE%\certs\ca-bundle.crt
# or *nix
$HOME/certs/cabundle.crt
Thanks to all the brilliant answers in:
How to get response SSL certificate from requests in python?
I have put together the following to attempt to take it a step further.
https://github.com/neozenith/get-ca-py
Set the configuration in pip and conda so that it knows where this CA store resides with our extra self-signed CA.
pip config set global.cert %USERPROFILE%\certs\ca-bundle.crt
conda config --set ssl_verify %USERPROFILE%\certs\ca-bundle.crt
OR
pip config set global.cert $HOME/certs/ca-bundle.crt
conda config --set ssl_verify $HOME/certs/ca-bundle.crt
THEN
pip config list
conda config --show ssl_verify
# Hot tip: use -v to show where your pip config file is...
pip config list -v
# Example output for macOS and homebrew installed python
For variant 'global', will try loading '/Library/Application Support/pip/pip.conf'
For variant 'user', will try loading '/Users/jpeak/.pip/pip.conf'
For variant 'user', will try loading '/Users/jpeak/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/pip.conf'