Logstash make a copy a nested field with mutate.add_field

Question

I wanted to make a copy of a nested field in a Logstash filter but I can't figure out the correct syntax. Here is what I try:

incorrect syntax:

mutate {
    add_field => { "received_from" => %{beat.hostname} }
}

beat.hostname is not replaced

mutate {
    add_field => { "received_from" => "%{beat.hostname}" }
}

beat.hostname is not replaced

mutate {
    add_field => { "received_from" => "%{[beat][hostname]}" }
}

beat.hostname is not replaced

mutate {
    add_field => { "received_from" => "%[beat][hostname]" }
}

No way. If I give a non nested field it works as expected.

The data structure received by logstash is the following:

{
       "@timestamp" => "2016-08-24T13:01:28.369Z",
             "beat" => {
                "hostname" => "etg-dbs-master-tmp",
                "name" => "etg-dbs-master-tmp"
    },
            "count" => 1,
               "fs" => {
        "device_name" => "/dev/vdb",
              "total" => 5150212096,
               "used" => 99287040,
             "used_p" => 0.02,
               "free" => 5050925056,
              "avail" => 4765712384,
              "files" => 327680,
         "free_files" => 326476,
        "mount_point" => "/opt/ws-etg/datas"
    },
             "type" => "filesystem",
         "@version" => "1",
             "tags" => [
                [0] "topbeat"
              ],
      "received_at" => "2016-08-24T13:01:28.369Z",
    "received_from" => "%[beat][hostname]"
}

Answer 1

EDIT:

Since you didn't show your input message I worked off your output. In your output the field you are trying to copy into already exists, which is why you need to use replace. If it does not exist, you do in deed need to use add_field. I updated my answer for both cases.

EDIT 2: I realised that your problem might be to access the value that is nested, so I added that as well :)

you are using the mutate filter wrong/backwards.

First mistake:

You want to replace a field, not add one. In the docs, it gives you the "replace" option. See: https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-replace

Second mistake, you are using the syntax in reverse. It appears that you believe this is true:

"text I want to write" => "Field I want to write it in" 

While this is true:

"myDestinationFieldName" => "My Value to be in the field" 

With this knowledge, we can now do this:

mutate {
    replace => { "[test][a]" => "%{s}"}
}

or if you want to actually add a NEW NOT EXISTING FIELD:

mutate {
        add_field => {"[test][myNewField]" => "%{s}"}
}

Or add a new existing field with the value of a nested field:

mutate {
        add_field =>  {"some" => "%{[test][a]}"}
}

Or more details, in my example:

input {
  stdin {
  }
}

filter {
    json {
        source => "message"
    }

    mutate {
        replace => { "[test][a]" => "%{s}"}
        add_field => {"[test][myNewField]" => "%{s}"}
        add_field => {"some" => "%{[test][a]}"}
    }
}

output {
          stdout { codec => rubydebug }
}

This example takes stdin and outputs to stdout. It uses a json filter to parse the message, and then the mutate filter to replace the nested field. I also add a completely new field in the nested test object. And finally creates a new field "some" that has the value of test.a

So for this message:

{"test" : { "a": "hello"}, "s" : "to_Repalce"}

We want to replace test.a (value: "Hello") with s (Value: "to_Repalce"), and add a field test.myNewField with the value of s.

On my terminal:

artur@pandaadb:~/dev/logstash$ ./logstash-2.3.2/bin/logstash -f conf2/
Settings: Default pipeline workers: 8
Pipeline main started
{"test" : { "a": "hello"}, "s" : "to_Repalce"}
{
   "message" => "{\"test\" : { \"a\": \"hello\"}, \"s\" : \"to_Repalce\"}",
  "@version" => "1",
"@timestamp" => "2016-08-24T14:39:52.002Z",
      "host" => "pandaadb",
      "test" => {
             "a" => "to_Repalce",
    "myNewField" => "to_Repalce"
},
         "s" => "to_Repalce"
         "some" => "to_Repalce"
}

The value has succesfully been replaced.

A field "some" with the replaces value has been added

A new field in the nested array has been added.

if you use add_field, it will convert a into an array and append your value there.

Hope this solves your issue,

Artur