java.net.SocketException: Connection reset With HTTPConnection

Question

I am trying to hit some external API to fetch some data. When the data size is small, everything works fine but when the size of data returned by the API is big I get CONNECTION RESET exception. The below code is from java class InterfaceHelper and I have also marked the comment at the line no where I am getting exception [Its while trying to read data from InputStream].

I have tried to search so many question on STACKOVERFLOW itself but didn't find an appropriate answer. So please don't mark it as duplicate question. I want to know what is the reason behind this problem, and what is the proper solution for this problem. If you find this question as duplicate please answer it before marking it as duplicate. I dare you.

Find below my code which I have used. URL is some dummy url as for security reason I cannot mention the actual URL I have used.

 try{
        URL url = new URL("http://example.com/someParams/SOME-ACCESS-TOKEN");
        HttpURLConnection connection = (HttpURLConnection)url.openConnection();
        connection.setRequestMethod("GET");
        connection.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
        connection.setRequestProperty("Content-Language", "en-US");
        connection.setRequestProperty("X-EXAMPLE-LOGIN", "XXXXXXXX");
        connection.setRequestProperty("X-EXAMPLE-PASSWORD", "XXXXXX");
        connection.setUseCaches(false);
        connection.setDoInput(true);
        connection.setDoOutput(true);
        DataInputStream input = new DataInputStream(connection.getInputStream());
        String  ret = "";
        if(input!=null){
            for( int c = input.read(); c != -1; c = input.read() ) { //InterfaceHelper.java:695
                ret = ret + String.valueOf((char)c);
            }
        }

         if(input!=null)
             input.close();
         if(connection!=null)
             connection.disconnect();

                if(ret!=null && ret.length()>0){
                    return ret;
                }

    }catch(Exception e) {
        e.printStackTrace();
    } 

This the exception I get

        java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:196)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
        at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
        at sun.security.ssl.InputRecord.read(InputRecord.java:509)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        at sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)
        at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:689)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3053)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3047)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3035)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at com.lsa.akosha.util.InterfaceHelper.hitApiBrandWatch(InterfaceHelper.java:695)
        at com.lsa.akosha.service.brand.BrandCronService.brandSentiments(BrandCronService.java:288)
        at com.lsa.akosha.util.thread.BrandWatchCronConverse.executeInternal(BrandWatchCronConverse.java:60)
        at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)

Answer 1

maybe its just quirks, but I've found some "weird" things in your post.

1. You use SSL connection (as it appears in the stacktrace) but in the provided example this is no HTTPS at all.
2. application/x-www-form-urlencoded header is usually used with POST requests. Full explanation on this read here

I'm just saying that this can be misleading to some extent.

Now regarding the question itself.

All-in-all - connection reset means that for some reason the connection between the client and the server is broken. For example, the server decides to close the connection. The debugging process can be really tricky here, so if I were you, I would have tried a couple of techniques, depending on environment you're working on:

1. Remove SSL and work with plain HTTP. This way you can put a proxy in-between and analyze the network traffic better. Stuff like Burp can help here.

2. Try to eliminate the possibility that some firewall/proxy/whatever just dumps the connection for its own reasons. Maybe it worth to run this code in the same machine with the server (of course if its a viable option) just for the sake of testing how this works with "localhost", you know.

3. I'm not really familiar with this fairly low level API and know all its quirks. But maybe you can use HttpClient instead, this way you'll probably eliminate the need to know all the "low-level" flags, maybe its something wrong there.

Hope this helps