failed connect or "certificate verify failed" on LWP HTTPS GET

Question

I posted this problem on Perl Monks yesterday but it worked for everyone who tried it (see http://www.perlmonks.org/?node_id=909968). However, I was using a different URL hoping to simplify the problem.

I'm attempting to connect to api.betfair.com via HTTPS and they have a valid certificate which I've verified in my browser. I am running ubuntu and have 2 versions of Perl. The system one 5.10.0 works and 5.14.0 installed via perlbrew fails. The code is:

use LWP::UserAgent; 
use strict;
use warnings;

#$ENV{HTTPS_CA_FILE} = "/usr/share/ca-certificates/cacert.org/cacert.org.crt";

my $ua  = LWP::UserAgent->new; 
my $req = HTTP::Request->new(GET => 'https://api.betfair.com');
my $res = $ua->request($req);

print $res->headers_as_string;
print $res->content;

Running this under the system Perl 5.10.0 it works fine and I get:

Date: Fri, 17 Jun 2011 08:33:04 GMT
Accept-Ranges: bytes
ETag: W/"0-1307353787000"
Content-Length: 0
Content-Type: text/html
Last-Modified: Mon, 06 Jun 2011 09:49:47 GMT
Client-Date: Fri, 17 Jun 2011 08:33:04 GMT
Client-Peer: 84.20.200.10:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
Client-SSL-Cert-Subject: /C=GB/ST=London/L=London/O=The Sporting Exchange Ltd/OU=IS/OU=Terms of use at www.verisign.com/rpa (c)05/CN=*.betfair.com
Client-SSL-Cipher: RC4-MD5
Set-Cookie: NSC_mc-80-qvcbqj.efgbvmu=ffffffff09208c5545525d5f4f58455e445a4a4229a0;expires=Fri, 17-Jun-2011 20:33:05 GMT;path=/;httponly

Running it under Perl 5.14.0 I get: Content-Type: text/plain Client-Date: Fri, 17 Jun 2011 08:34:30 GMT Client-Warning: Internal response Can't connect to api.betfair.com:443

If I uncomment the setting of HTTPS_CA_FILE and rerun in 5.14.0 I get:

Content-Type: text/plain
Client-Date: Fri, 17 Jun 2011 08:35:09 GMT
Client-Warning: Internal response
Can't connect to api.betfair.com:443 (certificate verify failed)

LWP::Protocol::https::Socket: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /home/martin/perl5/perlbrew/perls/perl-5.14.0/lib/site_perl/5.14.0/LWP/Protocol/http.pm line 51.

I have Mozilla::CA installed at version 20110409. Mozilla::CA::SSL_ca_file() returns "/home/martin/perl5/perlbrew/perls/perl-5.14.0/lib/site_perl/5.14.0/Mozilla/CA/cacert.pem" and it exists and is readable by me. I am using LWP 6.02 in Perl 5.14.0 and 5.836 in Perl 5.10.0. I read setting HTTPS_DEBUG=1 should output some debug info but it only does this (for me) when using Perl 5.10.0 and not 5.14.0.

I'm not an SSL guru by any means but I tried some things I found and they just make me more confused:

openssl verify -verbose -CAfile /home/martin/perl5/perlbrew/perls/perl-5.14.0/lib/site_perl/5.14.0/Mozilla/CA/cacert.pem < /dev/null
unable to load certificate
10888:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:647:Expecting: TRUSTED CERTIFICATE

openssl s_client -CAfile /usr/local/share/perl/5.10.0/Mozilla/CA/cacert.pem -connect api.betfair.com:443 < /dev/null
CONNECTED(00000003)
depth=1 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=GB/ST=London/L=London/O=The Sporting Exchange Ltd/OU=IS/OU=Terms of use at www.verisign.com/rpa (c)05/CN=*.betfair.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority - G2/OU=(c) 1998 VeriSign, Inc. - For authorized use only/OU=VeriSign Trust Network
---
Server certificate
-----BEGIN CERTIFICATE-----
certificate snipped
sg==
-----END CERTIFICATE-----
subject=/C=GB/ST=London/L=London/O=The Sporting Exchange Ltd/OU=IS/OU=Terms of use at www.verisign.com/rpa (c)05/CN=*.betfair.com
issuer=/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
---
No client certificate CA names sent
---
SSL handshake has read 3068 bytes and written 303 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-MD5
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-MD5
    Session-ID: 81802384A47AF45D2D809A2D10041A4E0B4B4DD821507569216A199ED467B207
    Session-ID-ctx: 
    Master-Key: 50DEC11CD2FA57E9BFA95B0156905D2717A79F333A2028FCCCB0F1C32A6B35202A958CEF24D3D2332A00CDCD158B40FB
    Key-Arg   : None
    Start Time: 1308304989
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
DONE

UPDATE: I thought it was because I had PERL_UNICODE=SAL set but unsetting this does not fix the problem.

UPDATE: versions: Linux ubuntu 10.10 codename maverick openssl 0.9.80 (I believe up to date on my ubuntu distribution

Answer 1

$ openssl s_client -connect api.betfair.com:443 < /dev/null > api.betfair.com.pem
$ openssl x509 -in api.betfair.com.pem -issuer_hash
eb99629b

Well, whaddayasay, it's the same stupid intermediate certificate 0xeb99629b I've seen missing before with other people, see comment above for details and how to get it.

Out of curiosity, what version of OpenSSL and ca-certificates are you running? What's your system version/vendor?