Getting KafkaConsumer is not safe for multi-threaded access error when I use CuratorFrameworkFactory.newClient()

java multithreading apache-kafka apache-curator
edrichhans picture edrichhans · Dec 19, 2018 · Viewed 8.6k times · Source

I am receiving this error below:

java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

when I include this in my code:

CuratorFrameworkFactory.newClient()

I don't understand what's causing the error. Any help?

Thanks.

Answer

Sammy picture Sammy · Dec 19, 2018

Kafka Javdocs clearly mentions that :

The Kafka consumer is NOT thread-safe. All network I/O happens in the thread of the application making the call. It is the responsibility of the user to ensure that multi-threaded access is properly synchronized. Un-synchronized access will result in ConcurrentModificationException.

More details present at http://kafka.apache.org/21/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#multithreaded

Related questions

Kafka consumer for multiple topic

I have a list of topics (for now it's 10) whose size can increase in future. I know we can spawn multiple threads (per topic) to consume from each topic, but in my case if the number of topics increases, then …

Read More
How to use multi-thread consumer in kafka 0.9.0?

The doc of kafka give an approach about with following describes: One Consumer Per Thread:A simple option is to give each thread its own consumer > instance. My code: public class KafkaConsumerRunner implements Runnable { private final AtomicBoolean closed = new …

Read More
How to write Kafka consumers - single threaded vs multi threaded

I have written a single Kafka consumer (using Spring Kafka), that reads from a single topic and is a part of a consumer group. Once a message is consumed, it will perform all downstream operations and move on to the …

Read More