I'm trying to use Java's HttpURLConnection to do a "conditional get", but I never get a 304 status code

Question

Here is my code:

    final HttpURLConnection conn = (HttpURLConnection) sourceURL.openConnection();
    if (cachedPage != null) {
        if (cachedPage.eTag != null) {
            conn.setRequestProperty("If-None-Match", cachedPage.eTag);
        }
        conn.setIfModifiedSince(cachedPage.pageLastModified);
    }

    conn.connect();

    if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {

        newCachedPage.eTag = conn.getHeaderField("ETag");
        newCachedPage.pageLastModified = conn.getHeaderFieldDate("Last-Modified", 0);

    } else if (conn.getResponseCode() == HttpURLConnection.HTTP_NOT_MODIFIED) {
        // Never reaches here
    }

I never seem to get the HTTP_NOT_MODIFIED response code, even hitting the same server several times in quick succession - where there is definitely no change to the page. Also, conn.getHeaderField("ETag") always seems to respond null, and sometimes conn.getHeaderFieldDate("Last-Modified", 0) returns 0. I've tried this against a variety of web servers.

Can anyone tell me what I'm doing wrong?

Answer 1

You're all dependent on the server config.

If you get an Expires response header, then it just means that you don't need to request anything until the specified expire time. If you get a Last-Modified response header, then it means that you should be able to use If-Modified-Since to test it. If you get an ETag response header, then it means that you should be able to use If-None-Match to test it.

Lets take http://cdn3.sstatic.net/stackoverflow/img/favicon.ico as an example (the Stackoverflow's favicon image):

URLConnection connection = new URL("http://cdn3.sstatic.net/stackoverflow/img/favicon.ico").openConnection();
System.out.println(connection.getHeaderFields());

This gives:

{null=[HTTP/1.1 200 OK], ETag=["9d9bd8b1165cb1:0"], Date=[Wed, 17 Aug 2011 17:57:07 GMT], Content-Length=[1150], Last-Modified=[Wed, 06 Oct 2010 02:53:46 GMT], Content-Type=[image/x-icon], Connection=[keep-alive], Accept-Ranges=[bytes], Server=[nginx/0.8.36], X-Cache=[HIT], Cache-Control=[max-age=604800]}

Now, do a If-Modified-Since with the same value as Last-Modified:

URLConnection connection = new URL("http://cdn3.sstatic.net/stackoverflow/img/favicon.ico").openConnection();
connection.setRequestProperty("If-Modified-Since", "Wed, 06 Oct 2010 02:53:46 GMT");
System.out.println(connection.getHeaderFields());

This gives as expected a 304:

{null=[HTTP/1.1 304 Not Modified], ETag=["9d9bd8b1165cb1:0"], Date=[Wed, 17 Aug 2011 17:57:42 GMT], Last-Modified=[Wed, 06 Oct 2010 02:53:46 GMT], Connection=[keep-alive], Server=[nginx/0.8.36], X-Cache=[HIT], Cache-Control=[max-age=604800]}

Now, do a If-None-Match with the same value as ETag:

URLConnection connection = new URL("http://cdn3.sstatic.net/stackoverflow/img/favicon.ico").openConnection();
connection.setRequestProperty("If-None-Match", "9d9bd8b1165cb1:0");
System.out.println(connection.getHeaderFields());

This gives unexpectedly a 200:

{null=[HTTP/1.1 200 OK], ETag=["9d9bd8b1165cb1:0"], Date=[Wed, 17 Aug 2011 18:01:42 GMT], Content-Length=[1150], Last-Modified=[Wed, 06 Oct 2010 02:53:46 GMT], Content-Type=[image/x-icon], Connection=[keep-alive], Accept-Ranges=[bytes], Server=[nginx/0.8.36], X-Cache=[HIT], Cache-Control=[max-age=604800]}

Even more surprising, when the both headers are set with random garbage value as ETag, the server still gives a 304. This is an indication that the If-None-Match is completely ignored by the server behind http://cdn3.sstatic.net. That might be a (proxy) configuration issue or be done fully awarely (not for obvious reasons imho).