Send KafkaProducer from local machine to hortonworks sandbox on virtualbox

Question

I have a really simple producer that I am running through eclipse on my windows local machine... What I really want is to get a message through to kafka, so I will be able to view the broker through zookeeper. Just to see how communication works from end to end... So here goes the code:

Properties props = new Properties();
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9020");
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());

    KafkaProducer<String,String> producer = new KafkaProducer<String,String>(props);

    boolean sync = true;
    String topic="mytopic";
    String key = "mykey";
    String value = "myvalue";

    ProducerRecord<String,String> producerRecord = new ProducerRecord<String,String>(topic, key, value);

    if (sync) {
        producer.send(producerRecord).get();
    } else {
        producer.send(producerRecord);
    }

    producer.close();

However after some time I get

Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:437)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:352)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248)
at kafkaProducer.TestProducer.main(TestProducer.java:30)  Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

I have hortonworks sandbox setup, with kafka running but I cannot seem to connect to it. I tried port forwarding in the virtualbox network configurations but still have the same issue. Is there something that I am missing?

Answer 1

You are correct in opening the ports, presumably 9092 and 2181 by default if you're also trying to create/use a Kafka consumer. Kafka however advertises its "hostname" to producers and consumers to use, that name needs to resolve from where you're connecting. The VirtualBox/VM host name does not since there's no entry on the hosting machine that tells it to resolve the hostname and go to localhost in your case and there's nothing that helps resolve it like a DNS. You could edit your hosts file but that's too intrusive. Long story short, Kafka acknowledges this might be a requirement and provides a configuration parameter for you to override what Kafka tells the world about how to get to the broker. Config is called advertised.host.name and advertised.port. Unless you're changing the port you will only need to set the advertised.host.name.

Try setting advertised.host.name in Kafka's server.properties configuration file to localhost. This along with opening the ports should do the trick.