Back pressure in Kafka

apache-kafka spark-streaming backpressure
sirigiri sai kumar picture sirigiri sai kumar · Apr 5, 2018 · Viewed 11.3k times · Source

I have a situation in Kafka where the producer publishes the messages at a very higher rate than the consumer consumption rate. I have to implement the back pressure implementation in kafka for further consumption and processing.

Please let me know how can I implement in spark and also in normal java api.

Answer

Robin Moffatt picture Robin Moffatt · Apr 5, 2018

Kafka acts as the regulator here. You produce at whatever rate you want to into Kafka, scaling the brokers out to accommodate the ingest rate. You then consume as you want to; Kafka persists the data and tracks the offset of the consumers as they work their way through the data they read.

Related questions

How to write spark streaming DF to Kafka topic

I am using Spark Streaming to process data between two Kafka queues but I can not seem to find a good way to write on Kafka from Spark. I have tried this: input.foreachRDD(rdd => rdd.foreachPartition(partition => …

Read More
IBM MQ versus Apache Kafka

I am designing a new architecture big data where my client has as IBM MQ broker. We use to work with Kafka in big data architectures, because I suppose it is faster, reliable and stronger than the alternatives, and because …

Read More
Limit Kafka batches size when using Spark Streaming

Is it possible to limit the size of the batches returned by the Kafka consumer for Spark Streaming? I am asking because the first batch I get has hundred of millions of records and it takes ages to process and …

Read More