Running kafka connect in Distributed mode?

apache-kafka kafka-consumer-api apache-kafka-connect confluent-platform
Pavan_Obj picture Pavan_Obj · Mar 6, 2018 · Viewed 8.7k times · Source

I have a total of 3 VM's(CloudVPS). Each of them has java, confluent open source installed on them. In VM1 I am running 3 processes of Splunk-sink-connector which reads from different topics and are running on different ports. And using REST calls I posted JSON configuration to each of them.

Since I am running in distributed mode I want to take advantage of other 2 VM's also. Can anyone please tell me what to do, to add other 2 VM's to those 3 processes to achieve parallel processing.

Answer

Robin Moffatt picture Robin Moffatt · Mar 6, 2018

You just need to run Kafka Connect in Distributed mode on the three VMs, follow the instructions here and make sure you give them all the same group.id which identifies them as members of the same cluster (and thus eligible for sharing workload of tasks out across them). More config details for distributed mode here.

See also:

Related questions

How to connect Kafka with Elasticsearch?

I am new in Kafka, I use kafka to collect netflow through logstash(it is ok), and I want to send the data to elasticsearch from kafka, but there are some problems. My question is how can I connect Kafka …

Read More
How can i get all group list with kafka0.10.x

when i use kafka0.8.x，i can get all group list with zookeeper by path /consumers/group_id, because offsets will commit to zookeeper in 0.8.x. But in version of 0.10.x, offsets commited to kafka broker like a path /brokers/…

Read More
Kafka Connect JDBC sink connector not working

I am trying to use Kafka Connect JDBC sink connector to insert data into Oracle but it is throwing an error . I have tried with all the possible configurations of the schema. Below is the examples . Please suggest if I …

Read More