Spring RestTemplate Large Files ContentLength Auto Changed

Question

Got a question on RestTemplate with Large Files. Is there a max content length and/or does it automatically get changed despite explicitly being set?

I have a file that is 5GB that I want to send to my CDN (Akamai) via HTTP PUT command. I've set it up like below (to avoid java heap space errors), however I noticed that when the command is executed the Content-Length gets changed.

    @Autowired
    @Qualifier("restTemplateUpload")
    public void setRestTemplateUpload(RestTemplate restTemplateUpload)
    {
        this.restTemplateUpload = restTemplateUpload;
        SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
        requestFactory.setBufferRequestBody(false);
        restTemplateUpload.setRequestFactory(requestFactory);
    }

And the call to send the file:

    Resource resource = new FileSystemResource(localFile);

    HttpHeaders headers = new HttpHeaders();
    Map<String, String> headerValues = new HashMap<String, String>();
    headerValues.put("X-Akamai...", value); //There are more headers, but you can assume this is correctly done as I've been able to transmit smaller files
    headerValues.put("Host", host);
    headers.setAll(headerValues);
    headers.setContentType(MediaType.MULTIPART_FORM_DATA);


    //I've tried using this and not using this but the result is
    //the same, the request sent changes the value to 1469622184
    headers.setContentLength(resource.contentLength()); 


    System.out.println(headers.getContentLength()); //Result is: 5764589480
    HttpEntity<Resource> uploadRequest = new HttpEntity<Resource>(resource, headers);
    response = restTemplateUpload.exchange(hostPath, HttpMethod.PUT, uploadRequest, String.class);

Charles Proxy reports Content-Length 1469622184 and the connection gets killed when the request reaches that length.

Java reports the error:

org.springframework.web.client.ResourceAccessException: I/O error on PUT request for "http://hidden.com/path/largefile.txt": too many bytes written;"

EDIT:

Some more observations, when I hardcode the Content-Length like so: headerValues.put("Content-Length", "3764589480"); The Content-Length field actually doesn't get sent and instead the header Transfer-Encoding: chunked gets sent.

So I ran more tests and this is what happens:

Hardcoded Content Length: Result in Charles

"1064589480": Sends Content-Length: 1064589480

"2064589480": Sends Content-Length: 2064589480

"3064589480": Removes Content-Length, Adds Transfer-Encoding: chunked

"3764589480": Removes Content-Length, Adds Transfer-Encoding: chunked

"4294967295": Removes Content-Length, Adds Transfer-Encoding: chunked

"4294967296": Sends Content-Length: "0", Immediate Crash, too many bytes written

"4764589480": Sends Content-Length: "469622184"

"5764589480": Sends Content-Length: "1469622184"

As far as I can deduce, at a certain number, RestTemplate will switch from Content-Length to Transfer-Encoding. Once it passes 4294967295, it starts passing Content-Length again back to 0. My best resolution for this is to never specify a value higher than 4294967295. If it's above that threshold, then simply set it to 4294967295 and RestTemplate will change it to Transfer-Encoding: chunked anyhow. Is there some kind of overflow/precision point issue?

Answer 1

AFAIK, 4294967295 bytes is the maximum number you can put in the Content Length field. Even if it is over this amount, I'd recommend setting it to 4294967295 as RestTemplate will automatically switch from using Content-Length to using Transfer-Encoding: chunked so it won't matter what size you put anyhow.

public static final String MAX_TRANSFER_SIZE = "4294967295";

...

HttpHeaders headers = new HttpHeaders();
Map<String, String> headerValues = new HashMap<String, String>();
if (resource.contentLength() > Long.parseLong(MAX_TRANSFER_SIZE))
{
        // This is as high as the content length can go. At this size,
        // RestTemplate automatically converts from using Content-Length to
        // using Transfer-Encoding:chunked.
        headerValues.put("Content-Length", MAX_TRANSFER_SIZE);
}
headers.setAll(headerValues);