Bulk loading JSON object as document into elasticsearch

json curl elasticsearch elasticsearch-bulk-api
codeBarer picture codeBarer · Sep 5, 2014 · Viewed 10k times · Source

Is there a way to bulk load the data below into elasticsearch without modifying the original content? I POST each object to be a single document. At the moment I'm using Python to parse through individual objects and POST them one at a time. 

{
   {"name": "A"},
   {"name": "B"},
   {"name": "C"},
   {"name": "D"},
}

Doing this type of processing in production from REST servers into elasticsearch is taking a lot of time.

Is there a single POST/curl command that can upload the file above at once and elasticsearch parses it and makes each object into its own document?

We're using elasticsearch 1.3.2

Answer

coffeeaddict picture coffeeaddict · Sep 5, 2014

Yes, you can do bulk api via curl by using the _bulk endpoint. But not custom parsing. Whatever process that creates the file can format it to ES specification if that is an option. See here:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html

There is also bulk support in python via helper. See here: http://elasticsearch-py.readthedocs.org/en/master/helpers.html

Related questions

How do I POST JSON data with cURL?

I use Ubuntu and installed cURL on it. I want to test my Spring REST application with cURL. I wrote my POST code at the Java side. However, I want to test it with cURL. I am trying to post …

Read More
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I am getting error Expecting value: line 1 column 1 (char 0) when trying to decode JSON. The URL I use for the API call works fine in the browser, but gives this error when done through a curl request. The following is …

Read More
Curl GET request with json parameter

I am trying to send a "GET" request to a remote REST API from Command Prompt via cURL like this: curl -X GET -H "Content-type: application/json" -H "Accept: application/json" "http://server:5050/a/c/getName/{"param0":"pradeep"}" But it …

Read More