How to write a file to Kafka Producer

file apache-kafka kafka-consumer-api kafka-producer-api
Katie picture Katie · Oct 22, 2015 · Viewed 62.9k times · Source

I am trying to load a simple text file instead of standard input in Kafka. After downloading Kafka, I performed the following steps:

Started zookeeper:

bin/zookeeper-server-start.sh config/zookeeper.properties

Started Server

bin/kafka-server-start.sh config/server.properties

Created a topic named "test":

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Ran the Producer:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 
Test1
Test2

Listened by the Consumer:

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
Test1
Test2

Instead of Standard input, I want to pass a data file or even a simple text file to the Producer which can be seen directly by the Consumer. Any help would really be appreciated. Thanks!

Answer

Balázs Németh picture Balázs Németh · Oct 22, 2015

You can pipe it in:

kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic
--new-producer < my_file.txt

Found here.

From 0.9.0:

kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic < my_file.txt

Related questions

How do I check whether a file exists without exceptions?

How do I check if a file exists or not, without using the try statement?

Read More
How to read a file line-by-line into a list?

How do I read every line of a file in Python and store each line as an element in a list? I want to read the file line by line and append each line to the end of the list.

Read More
How do I include a JavaScript file in another JavaScript file?

Is there something in JavaScript similar to @import in CSS that allows you to include a JavaScript file inside another JavaScript file?

Read More