How to check what Filebeat is sending to Logstash?

Question

I already have Logstash configured to directly pick up files (Gatling logs), process them using custom logic, and send to Elasticsearch. And it's working fine. I'm now trying to setup Filebeat to send the files to it instead.

The basics seem to work as I see the new entires ending up in Elasticsearch, but they look all wrong. My theory is that Logstash is configured to parse Gatling logs, but Filebeat doesn't send the logs directly, but some JSON or other format containing the metadata as well, and Logstash needs to be reconfigured to parse this instead.

But... how do I find out what Filebeat is sending? Is the format documented somewhere?

Answer 1

add the setting to filebeat.yml

you could see the output log as file in your path to see what happened.

output.file:
# Boolean flag to enable or disable the output module.
enabled: true

# Path to the directory where to save the generated files. The option is
# mandatory.
path: "your path"

# Name of the generated files. The default is `filebeat` and it generates
# files: `filebeat`, `filebeat.1`, `filebeat.2`, etc.
filename: filebeat

# Maximum size in kilobytes of each file. When this size is reached, and on
# every filebeat restart, the files are rotated. The default value is 10240
# kB.
#rotate_every_kb: 10000

# Maximum number of files under path. When this number of files is reached,
# the oldest file is deleted and the rest are shifted from last to first. The
# default is 7 files.
number_of_files: 7

if you want to see the network setting

please add the setting to see the log.

logging.level: warning
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/mybeat
  name: mybeat.log
  keepfiles: 7