Retry logic in kafka consumer

Question

I have a use case where i consume certain logs from a queue and hit some third party API's with some info from that log , in case the third party system is not responding properly i wish to implement a retry logic for that particular log .

I can add a time field and repush the message to the same queue and this message will get again consumed if its time field is valid i.e less than current time and if not then get pushed again into queue.

But this logic will add same log again and again until retry time is correct and the queue will grow unnecessarily.

Is there is better way to implement retry logic in Kafka ?

Answer 1

You can create several retry topics and push failed task there. For instance you can create 3 topics with different delays in mins and rotate the single failed task till the max attempt limit reached.

‘retry_5m_topic’ — for retry in 5 minutes

‘retry_30m_topic’ — for retry in 30 minutes

‘retry_1h_topic’ — for retry in 1 hour

See more for details: https://blog.pragmatists.com/retrying-consumer-architecture-in-the-apache-kafka-939ac4cb851a