Kafka producer send message expiring duo to 30003 ms has passed since last append

Question

I'm using Kafka 0.10.1.1 and write a test producer, but I get an error in producer.send onCompletion callback:

        producer.send(record, new Callback() {
        @Override
        public void onCompletion(RecordMetadata metadata, Exception e) {
            if (e != null){
                System.out.println(e.getMessage());
            }
        }
    });

Producer with Config: timeout.ms=30000, linger.ms=5, batch.size=1000

Got the following error message:

Expiring 1 record(s) for testtopic-2 due to 30004 ms has passed since last append

But this error message just appeared sometimes. The producer can work well sometimes.

Answer 1

Say a topic has 100 partitions (0-99). Kafka lets you produce records to a topic by specifying a particular partition. Faced the issue where I'm trying to produce to partition > 99, because brokers reject these records.

Its highly probable that your topic (testtopic) has less than 3 partition where as you are trying to produce to testtopic-2.