Is there any simulator/tool to generate messages for streaming?

streaming apache-kafka simulation messaging stub
Sameeksha Raina picture Sameeksha Raina · Oct 19, 2016 · Viewed 11.9k times · Source

For testing purpose, I need to simulate client for generating 100,000 messages per second and send them to kafka topic. Is there any tool or way that can help me generate these random messages?

Answer

serejja picture serejja · Oct 19, 2016

There's a built-in tool for generating dummy load, located in bin/kafka-producer-perf-test.sh (https://github.com/apache/kafka/blob/trunk/bin/kafka-producer-perf-test.sh). You may refer to https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/ProducerPerformance.java#L106 to figure out how to use it.

One usage example would be like that:

bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 10000000 --topic test --threads 10 --message-size 100 --batch-size 10000 --throughput 100000

The key here is the --throughput 100000 flag which indicated "throttle maximum message amount to approx. 100000 messages per second"

Related questions

Streaming via RTSP or RTP in HTML5

I'm building a web app that should play back an RTSP/RTP stream from a server http://lscube.org/projects/feng. Does the HTML5 video/audio tag support the rtsp or rtp? If not, what would the easiest solution be? …

Read More
Download/Stream file from URL - asp.net

I need to stream a file which will result in save as prompt in the browser. The issue is, the directory that the file is located is virtually mapped, so I am unable to use Server.MapPath to determine it's …

Read More
Streaming video from Android camera to server

I've seen plenty of info about how to stream video from the server to an android device, but not much about the other way, ala Qik. Could someone point me in the right direction here, or give me some advice …

Read More