GZIPInputStream - Corrupt GZIP Trailer

java stream gzip ioexception gzipinputstream
Paul Richter picture Paul Richter · Oct 12, 2013 · Viewed 8.4k times · Source

I have a static helper method responsible for getting a compressed JSON string from our Rails application, and decompressing the data before returning a String representation.

I wrote two JUnit tests that, one that tests that the JSON parses correctly, and another more basic test that determines whether a string of length greater than zero is returned from the server.

The problem: When I run the test suite, the first test method succeeds properly, and the other fails with an IOException and the message "Corrupt GZIP trailer" (see code below). I have determined that it is not the test itself that is failing, as the "successful" test is reversed when I cause the tests to run in opposite order (in other words, no matter what, it is always the second test that fails, no matter which of the two tests is run second).

This is the helper method:

public static String doHTTPGet(String urlString) throws IOException{
    URL weatherAPI = new URL(urlString);
    HttpURLConnection apiConnection = (HttpURLConnection) weatherAPI.openConnection();
    apiConnection.setRequestMethod("GET");
    apiConnection.setRequestProperty("Accept-Encoding", "gzip");

    apiConnection.connect();

    BufferedInputStream bufferedInputStream = new BufferedInputStream(apiConnection.getInputStream());
    byte[] inputByteBuffer = new byte[10 * 1024];
    ByteArrayOutputStream outputStream = new ByteArrayOutputStream(10 * 1024); // initialize the output stream with at least one buffer's worth of bytes
    while(bufferedInputStream.read(inputByteBuffer)  > -1){
        outputStream.write(inputByteBuffer);
    }

    outputStream.close();
    bufferedInputStream.close();
    apiConnection.disconnect();

    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(outputStream.toByteArray());
    byteArrayInputStream.close();

    GZIPInputStream gis = new GZIPInputStream(byteArrayInputStream);
    InputStreamReader inputStreamReader = new InputStreamReader(gis, "UTF-8");
    BufferedReader reader = new BufferedReader(inputStreamReader);

    String decompressedResponse = "";
    String line;

    // readLine() is generating the IOException on the second pass.
    while((line = reader.readLine()) != null){
        decompressedResponse += line;
    }

    reader.close();
    inputStreamReader.close();
    gis.close();

    return decompressedResponse;
}

The error occurs towards the bottom of the helper method, on the line while((line = reader.readLine()) != null).... Specifically, the error occurs on the reader.readLine().

And the two test methods:

@Test
public void testHttpGet(){
    try {
        // FILTERED_API_URL_WITH_TOKEN is merely the URL with an auth token
        String apiResponse = HTTPHelper.doHTTPGet(GlobalConstants.FILTERED_API_URL_WITH_TOKEN);
        assertNotNull(apiResponse);
        assertTrue("The size of the API response should be greater than zero. It is an empty string.", apiResponse.length() > 0);
    } catch (IOException e) {
        e.printStackTrace();
        assertTrue("An exception occured while trying to perform the HTTP Get to the api at URL " + GlobalConstants.FILTERED_API_URL_WITH_TOKEN, false);
    }
}

@Test
public void testAPIContent(){
    try {
        // the getAPIJson() method basically does the same as the testHttpGet
        // method, but converts the string to a json
        JSONObject jsonObject = XMLProducerFromAPI.getAPIJson();
        System.out.println(jsonObject);
        assertNotNull(jsonObject);
    } catch (IOException e) {
        e.printStackTrace();
        assertTrue("An IOException occured. See stack trace", false);
    } catch (JSONException e) {
        e.printStackTrace();
        assertTrue("A JSONException occured. See stack trace", false);
    }
}

I have read through this question and the answer, but I don't believe its applicable, (or maybe it is and I misunderstood, let me know if that is the case), and I tried their approach and only received the same message.

As the doHTTPGet method is static, and the objects created are done so within the body of the method, nothing (streams, connection objects, etc) should be reused. Frankly, I'm stumped.

Question: Have I done something wrong in my helper code, or have I misunderstood some usage of some object, that would yield the the "Corrupt GZIP Trailer" message? In short, what would cause this error in my scenario?

As always, please let me know if I've left anything out of this question.

EDIT

This is the stack trace: 

java.io.IOException: Corrupt GZIP trailer
    at java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:200)
    at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:92)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
    at java.io.InputStreamReader.read(InputStreamReader.java:167)
    at java.io.BufferedReader.fill(BufferedReader.java:136)
    at java.io.BufferedReader.readLine(BufferedReader.java:299)
    at java.io.BufferedReader.readLine(BufferedReader.java:362)
    at com.weathertx.xmlserver.support.HTTPHelper.doHTTPGet(HTTPHelper.java:60)
    at com.weathertx.xmlserver.tests.HttpHelperTest.getAPIResponse(HttpHelperTest.java:47)
    at com.weathertx.xmlserver.tests.HttpHelperTest.testHttpGet(HttpHelperTest.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

Answer

Paul Richter picture Paul Richter · Oct 18, 2013

Problem has been solved. To be frank, I don't quite understand why it would not work originally, or what was wrong with it (other than apparently being overly and unnecessarily complicated). Thanks to this solution, which I somehow missed my first search through, I was able to solve the problem by basically implementing more or less exactly what they did. This is my final code:

public static String doHTTPGet(String urlString) throws IOException{
    URL weatherAPI = new URL(urlString);
    HttpURLConnection apiConnection = (HttpURLConnection) weatherAPI.openConnection();
    apiConnection.setRequestMethod("GET");
    apiConnection.setRequestProperty("Accept-Encoding", "gzip");

    apiConnection.connect();

    InputStream gzippedResponse = apiConnection.getInputStream();
    InputStream decompressedResponse = new GZIPInputStream(gzippedResponse);
    Reader reader = new InputStreamReader(decompressedResponse, "UTF-8");
    StringWriter writer = new StringWriter();

    char[] buffer = new char[10240];
    for(int length = 0; (length = reader.read(buffer)) > 0;){
        writer.write(buffer, 0, length);
    }

    writer.close();
    reader.close();
    decompressedResponse.close();
    gzippedResponse.close();
    apiConnection.disconnect();

    return writer.toString();
}

So ultimately, I did not need to pass the data through byte array streams and all over the place. Other than my original approach being convoluted, if anybody knows specifically why my original algorithm yielded the "Corrupted GZIP trailer" error message after the first call of this method, by all means let me know.

Related questions

How do I read / convert an InputStream into a String in Java?

If you have a java.io.InputStream object, how should you process that object and produce a String? Suppose I have an InputStream that contains text data, and I want to convert it to a String, so for example I …

Read More
Get an OutputStream into a String

What's the best way to pipe the output from an java.io.OutputStream to a String in Java? Say I have the method: writeToStream(Object o, OutputStream out) Which writes certain data from the object to the given stream. However, …

Read More
Easy way to write contents of a Java InputStream to an OutputStream

I was surprised to find today that I couldn't track down any simple way to write the contents of an InputStream to an OutputStream in Java. Obviously, the byte buffer code isn't difficult to write, but I suspect I'm just …

Read More