Elasticsearch time-range query and data

datacruncher picture datacruncher · Aug 17, 2018 · Viewed 9.8k times · Source

I am struggling to formulate the right API search call for Elastic Search that will ask for ipv4address that I want in last 1 hour.

First attempt:

curl -X GET "localhost:9200/ipaddresses/_search" -H 'Content-Type: application/json' -d'
     {
       "query": {
         "match": {
           "ipv4address": {
             "query": "50.167.71.25"
           }
         }
       },
       "range": {
         "@timestamp": {
           "gte": "now-1h",
           "lt": "now"
         }
       }
     }
     '

{"error":{"root_cause":[{"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [range].","line":10,"col":12}],"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [range].","line":10,"col":12},"status":400}

Second attempt:

curl -X GET "localhost:9200/ipaddresses/_search" -H 'Content-Type: application/json' -d'
{
   "query": {
     "match": {
       "ipv4address": {
         "query": "50.167.71.25"
       }
     }
   },
   "fields": {
    "range": {
     "@timestamp": {
      "gte": "now-1h",
      "lt": "now"
     }
   }
  }
 }
 '

{"error":{"root_cause":[{"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [fields].","line":10,"col":14}],"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [fields].","line":10,"col":14},"status":400}

What I have in Kibana:

{
  "_index": "ipaddresses",
  "_type": "default",
  "_id": "TJdvR2UB9sEBYW4CrElF",
  "_version": 1,
  "_score": null,
  "_source": {
    "tags": [
      "blocked",
      "ipv4_address",
    ],
    "@version": "1",
    "@timestamp": "2018-08-17T10:30:25.118Z",
    "ipv4_metadata": {
      "host": "elk",
      "name": "blocks",
      "response_message": "OK",
      "code": 200,
      "times_retried": 0,
      "runtime_seconds": 0.066403,
      "response_headers": {
        "connection": "keep-alive",
        "x-frame-options": "sameorigin",
        "last-modified": "Fri, 17 Aug 2018 10:28:06 GMT",
        "keep-alive": "timeout=20",
        "date": "Fri, 17 Aug 2018 10:28:20 GMT",
        "content-type": "text/plain; charset=UTF-8",
        "server": "nginx/1.12.2",
        "transfer-encoding": "chunked",
        "etag": "W/\"5c7c5-5739f03f2997f\"",
        "cache-control": "public"
      }
    },
    "ipv4address": "50.167.71.25",
    "message": "50.167.71.25"
  },
  "fields": {
    "@timestamp": [
      "2018-08-17T10:30:25.118Z"
    ]
  },
  "sort": [
    1534501825118
  ]
}

What is wrong with the query? What if I would like to look as well for a "tag" field that equals blocked?

Please help me to connect the dots.

Answer

TechnocratSid picture TechnocratSid · Aug 17, 2018

This query will return the documents from last 1 hr:

{
   "query": {
     "range": {
       "@timestamp": {
         "gte": "now-1h",
         "lt": "now"
       }
     }
   }
 }

This query will return the documents where tag is blocked and is from last 1hr:

{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "tags": "blocked"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-1h",
              "lte": "now"
            }
          }
        }
      ]
    }
  }
}

You can limit the data to be returned using _source.

This query will only return the ipv4address:

{
  "_source": "ipv4address", 
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "tags": "blocked"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-1h",
              "lte": "now"
            }
          }
        }
      ]
    }
  }
}

If you want to apply more queries have a look at this.