kafka producers are very slow

Question

I am new in Kafka and I have a question that I'm not able to resolve.

I have installed Kafka and Zookeeper in my own computer in Windows (not in Linux) and I have created a broker with a topic with several partitions (playing between 6 and 12 partitions).

When I create consumers, they works perfectly and read at good speed, but referring producer, I have created the simple producer one can see in many web sites. The producer is inside a loop and is sending many short messages (about 2000 very short messages).

I can see that consumers read the 2000 messages very quicly, but producer sends message to the broker at more or less 140 or 150 messages per second. As I said before, I'm working in my own laptop (only 1 disk), but when I read about millions of messages per second, I think there is something I forgot because I'm light-years far from that.

If I use more producers, the result is worse.

Is a question of more brokers in the same node or something like that? This problem have been imposed to me in my job and I have not the possibility of a better computer.

The code for creating the producer is

public class Producer {

    public void publica(String topic, String strKey, String strValue) {
        Properties configProperties = new Properties();
        configProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        configProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, LongSerializer.class.getName());
        configProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        KafkaProducer<String, String> producer = new KafkaProducer<String, String>(configProperties);
        ProducerRecord<String, String> rec = new ProducerRecord<String, String>(topic, strValue);
        producer.send(rec);
    }
}

and the code for sending messages is (partial):

Producer prod = new Producer();

for (int i = 0; i < 2000; i++)
{
    key = String.valueOf(i);
    prod.publica("TopicName", key, texto + " - " + key);
    // System.out.println(i + " - " + System.currentTimeMillis());
}

Answer 1

You may create your Kafka producer once and use it every time you need to send a message:

public class Producer {
    private final KafkaProducer<String, String> producer; // initialize in constructor

    public void publica(String topic, String strKey, String strValue) {
        ProducerRecord<String, String> rec = new ProducerRecord<String, String>(topic, strValue);
        producer.send(rec);
    }
}

Also take a look at the producer and broker configurations available here. There are several options with which you can tune for your application's needs.