How Logstash is different than Kafka

apache-kafka devops elastic-stack
Chandan Gupta picture Chandan Gupta · Nov 29, 2016 · Viewed 11.1k times · Source

How Log stash is different than Kafka? and if both are same which is better? and How?

I found both are the pipelines where one can push the data for further processing.

Answer

Kulasangar picture Kulasangar · Nov 29, 2016

Logstash is a tool that can be used to collect, process and forward events and log messages. Collection is accomplished through a number of input plugins. You can use Kafka as an input plugin, where it will read events from a Kafka topic. Once an input plugin has collected data it can be processed by any number of filters which modify and annotate the event data. Finally events are routed to output plugins which can forward the events to a variety of external programs including Elasticsearch.

Where as Kafka is a messaging software that persists messages, has TTL, and the notion of consumers that pull data out of Kafka. Some of it's usages could be:

  • Stream Processing
  • Website Activity Tracking
  • Metrics Collection and Monitoring
  • Log Aggregation

So simply both of them have their own advantages and disadvantages. But then it depends on your requirements solely.

Related questions

List all kafka topics

I'm using kafka 0.10 without zookeeper. I want to get kafka topics list. This command is not working since we're not using zookeeper: bin/kafka-topics.sh --list --zookeeper localhost:2181. How can I get the same output without zookeeper?

Read More
Purge Kafka Topic

Is there a way to purge the topic in kafka? I pushed a message that was too big into a kafka message topic on my local machine, now I'm getting an error: kafka.common.InvalidMessageSizeException: invalid message size Increasing the …

Read More
Is there a way to delete all the data from a topic or delete the topic before every run?

Is there a way to delete all the data from a topic or delete the topic before every run? Can I modify the KafkaConfig.scala file to change the logRetentionHours property? Is there a way the messages gets deleted as …

Read More