Large file transfer from HTTP server running Java Jersey Rest API
I am writing a web service using Java JDK 1.7 and Jersey Web service Framework. One of the things I need to provide is a way to allow authenticated clients to download certain large data files ( 1-3 GB). Ideally I would like this to be a pause and resume type downloadable option. I tried the jersey multi-part API and was able to get it to work on my client machine upto 400 MB but beyond that it ran into out-of memory issues. I am also worried that the server might fail when faced with simultaneous download requests. Any thoughts on how this can be done? Is Netty an option? Any pointers on how Netty can be integrated into a existing Jersey based web service? Are there other frame works available to help accomplish this? I do have to use java for the web service. Any pointers will be helpful.
Answer
If you are getting stuck on out-of-memory issues, you should check how you are handling the data you are downloading. If you are using Jersey's ClientResponse, make sure you are using getEntityInputStream() and not getEntity(). This way, you can stream the data, write it to file, and toss it aside, rather than letting it build up in the Java heap space.
I can't really speak about your simultaneous download concerns, but if you are using the web services framework, then it should be handled properly.
For both issues, more info on your specific implementation, especially code, will help you get a better response.