How to handle huge data from a REST service

java web-services rest restful-url restful-architecture
om39a picture om39a · Feb 6, 2014 · Viewed 15.3k times · Source

We are using a REST service that returns huge data. In production, the server hardware can handle it. I need to test the REST service by getting the data on my local machine, which cannot handle the huge data. My local machine is a 4G i5. I am getting out of memory exception every time I hit the service.

response.getStatus() returns 200 status. But while collecting the data using input stream reader, I get an out of memory exception. 

BufferedReader br = new BufferedReader(new
InputStreamReader(newByteArrayInputStream(response.getEntity().getBytes())));

Is there any other way to collect the data without hitting the memory exception?

I tried pushing my VM size to 2G but, it still doesn't work.

Answer

Brian Kelly picture Brian Kelly · Feb 6, 2014

If the service supports it, use the HTTP standard approach of requesting chunks within the range of the entire response content payload.

Another approach (again, assuming the API supports it) is pagination.

Related questions

PUT and POST getting 405 Method Not Allowed Error for Restful Web Services

I am trying to set up a simple Restful Web-Service which returns either JSON or XML according to the Accept header. I am using Spring, Maven and WebLogic Server. I took the example from this post http://software.sawano.se/2012/03/…

Read More
Error 415 Unsupported Media Type: POST not reaching REST if JSON, but it does if XML

I am actually new to REST WS but really I don't get this 415 Unsupported Media Type. I am testing my REST with Poster on Firefox and the GET works fine for me, also the POST (when it's a application/xml) …

Read More
How to access parameters in a RESTful POST method

My POST method looks like this: @POST @Consumes({"application/json"}) @Path("create/") public void create(String param1, String param2){ System.out.println("param1 = " + param1); System.out.println("param2 = " + param2); } When I create a Jersey Client in Netbeans the method who …

Read More