Filebat/Logstash remove unwanted fields & values from output

Question

My Filebeat configuration is very simple -

- input_type: log
  paths:
    - C:\log\FilebeatInputTest.txt

output.logstash:
  hosts: ["http://X.X.X.X:XXXX"]

if I write something in ilebeatInputTest.txt like - This is from Filebeat

I get output in Elastic search something like - ....... "index": "logstash-" "source" : { "@timestamp": "2017-05-19T06:41:02.663Z", "beat": { "hostname": "CHITTARS02", "name": "CHITTARS02", "version": "5.4.0" }, "input_type": "log", "message": "This is from Filebeat", "offset": 23, "source": "C:\\log\\FilebeatInputTest.txt", "type": "log" } .....

My pipeline is Filebeat(monitoring FilebeatInputTest.txt) > Logstash > Elasticsearch

logstash.cnf as follows -

input {

    beats {
        port => 25000
    }
}
output {

    elasticsearch {
        hosts => ["http://xx.xx.xx.xx:XX"]
        user => "elastic"
        password => "changeme"
    }
}

Problem : Can I remove all unwanted keys & values from output? That is, I want my output should be something like -

....... "index": "logstash-" "source" : { "message": "This is from Filebeat", } ......

I want to remove "@timestamp", "beat","input_type""offset","source","type"

I tried with following -

filter{
    prune {
        blacklist_names => ["@timestamp", "beat","input_type""offset","source","type"]
    }

}

And

filter{
    mutate {
        remove_field => ["@timestamp", "beat","input_type""offset","source","type"]
    }
}

But no help, results are same

Answer 1

You're using the correct method, but there's a typo in your remove_field list. You missed a comma. It should be:

filter{
    mutate {
        remove_field => [ "@timestamp", "beat", "input_type", "offset", "source", "type" ]
    }
}