I am referring to these questions:
I used solution from second question to handle all suggested https protocols:
System.getProperties().setProperty("https.protocols", "TLSv1.2,TLSv1.1,TLSv1,SSLv3");
Full code:
/**
* Opens HTTP/HTTPS and setup connection to given URL.
*
* @return prepared HttpURLConnection connection
* @throws IOException in case URL is malformed or connection cannot be established
*/
public void openHttpUrlConnectionForGet() throws IOException {
// Set Https protocols
System.getProperties().setProperty("https.protocols", "TLSv1.2,TLSv1.1,TLSv1,SSLv3");
// Create connection
URL urlObject = new URL(location);
HttpURLConnection conn;
if (proxy != null) {
InetSocketAddress adr = new InetSocketAddress(proxy.getPk().getAddress(), proxy.getPk().getPort());
java.net.Proxy prx = new java.net.Proxy(java.net.Proxy.Type.HTTP, adr);
conn = (HttpURLConnection) urlObject.openConnection(prx);
} else {
conn = (HttpURLConnection) urlObject.openConnection();
}
conn.setRequestMethod("GET");
conn.setInstanceFollowRedirects(false);
// Setup SSL factory
if (conn instanceof HttpsURLConnection) {
HttpsURLConnection httpsc = (HttpsURLConnection) conn;
if (hostnameVerifier != null) {
httpsc.setHostnameVerifier(hostnameVerifier);
}
if (trustManager != null) {
try {
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, new TrustManager[]{trustManager}, new java.security.SecureRandom());
httpsc.setSSLSocketFactory(sc.getSocketFactory());
} catch (KeyManagementException | NoSuchAlgorithmException e) {
throw new RuntimeException("Cannot init HTTPS connection.", e);
}
}
}
// Configure timeouts
conn.setConnectTimeout(connectTimeout);
conn.setDefaultUseCaches(false);
conn.setUseCaches(false);
conn.setDoOutput(false);
conn.setReadTimeout(readTimeout);
// Check connection
if (StringUtils.isEmpty(userAgent)) {
throw new IllegalArgumentException("Cannot create HTTP(S) connection. User-Agent header is not set.");
}
if (StringUtils.isEmpty(accept)) {
throw new IllegalArgumentException("Cannot create HTTP(S) connection. Accept header is not set.");
}
if (StringUtils.isEmpty(acceptLanguage)) {
throw new IllegalArgumentException("Cannot create HTTP(S) connection. Accept-Language header is not set.");
}
if (StringUtils.isEmpty(acceptEncoding)) {
throw new IllegalArgumentException("Cannot create HTTP(S) connection. Accept-Encoding header is not set.");
}
// Set headers
conn.setRequestProperty("User-Agent", userAgent);
conn.setRequestProperty("Accept", accept);
conn.setRequestProperty("Accept-Language", acceptLanguage);
conn.setRequestProperty("Accept-Encoding", acceptEncoding);
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Upgrade-Insecure-Requests", "1");
if (cookies != null) {
conn.setRequestProperty("Cookie", cookies);
}
this.connection = conn;
}
However I still receive:
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at cz.wm.common.net.HttpConnection.retrieveContent(HttpConnection.java:73)
at cz.wm.agent.slave.job.search.AbstractSearchJob.getFinalURL(AbstractSearchJob.java:192)
at cz.wm.common.job.AbstractAgentJob.run(AbstractAgentJob.java:46)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
This are, for instance, the problematic sites that I can open via browser but can't open through java HttpConnection:
These are all sites which require the TLS SNI extension and fail otherwise. While Java 7+ provide this extension it does not add it in all cases. From https://javabreaks.blogspot.de/2015/12/java-ssl-handshake-with-server-name.html:
Whenever a custom HostNameVerifier is provided, java 8 fails to add the SNI extension header ...
It looks like your code does set a custom HostNameVerifier which is usually a bad idea anyway. Thus, either make sure your code does not set a custom HostNameVerifier or follow the workaround outlined in the provided link.