Kafka and firewall rules
We have a fairly strict network segmentation policy. I am using a cloud foundry instance to deploy an app to. The firewall rules have been set up to reach the kafka cluster from within the cloud foundry instance. I believe that the firewall rules have also been set up to get to the zookeeper instance as well. I need to actually confirm that one.
My problem seems to be that I can produce messages to kafka, but my consumer doesn't seem to be picking them up. It seems to hang while "polling".
Is there some hidden hosts or ports that I need to deal with for my firewall rules that are not just the standard hosts and ports to the kafka and zookeeper nodes?
Answer
Kafka and zookeeper are different things. If you are running both on the same machine, you need to open both ports, of corse.
kafka default ports:
- 9092, can be changed on server.properties;
zookeeper default ports:
- 2181 for client connections;
- 2888 for follower(other zookeeper nodes) connections;
- 3888 for inter nodes connections;
That's it.
Kafka, also has the listeners and advertised.listeners properties which grows some confusion on first users. To make it simple, listener is the network interface your server will bind, and advertised.listeners is the hostname or IP your server will register itself on zookeeper and listen to requests. If you put a hostname in there, your clients WILL have to use the hostname to connect. The advertised.listeners url is the one your clients will use to bootstrap the connection. Once connection is made, your client will get a connection to zookeeper to get other brokers urls. Your producer is not working because of that.
So, to make it work you need to open 2888 on your firewall too, not just 2181. And @Jaya Ananthram is wrong when he tells you that kafka needs 2181 port. It's a zookeeper port. The consumers on kafka 0.10 stills needs to contact zookeeper to persist some things, thats it.
Kafka 0.11.0.0 changed this and is making clients don't need zookeeper at all.