Can I use kafka over Internet?

authentication apache-kafka cluster-computing nat wan
ntysdd picture ntysdd · Sep 5, 2017 · Viewed 9k times · Source

Is kafka suitable for Internet-use?

More precisely, what I want is to expose kafka topics as "public interface", then external consumers (or producers) can connect to it. Is it possible?

I hear there are problems if I want to use the cluster in both internal and external networks, because it is then hard to configure advertised.host.name. Is that true?

And do I have to expose zookeeper as well? I think the new consumer/producer api no longer need that.

Answer

Hans Jespersen picture Hans Jespersen · Sep 5, 2017

Kafka's wire protocol is TCP-based and works fine over the public internet. In the latest versions of Kafka you can configure multiple interfaces for both internal and external traffic. Examples of Kafka over the internet in production include several Kafka-as-a-Service offerings from Heroku, IBM MessageHub, and Confluent Cloud.

You do not need to expose zookeeper if the Kafka clients use the new consumer API.

You may also choose to expose a REST Proxy such as the open source Confluent REST Proxy as a more client firewall friendly interface since it runs over HTTP(S) and will not be blocked by most corporate or personal firewalls.

Related questions

Kafka SASL zookeeper authentication

I am facing the following error while enabling SASL on Zookeeper and broker authentication. [2017-04-18 15:54:10,476] DEBUG Size of client SASL token: 0 (org.apache.zookeeper.server.ZooKeeperServer) [2017-04-18 15:54:10,476] ERROR cnxn.saslServer is null: cnxn object did not initialize its …

Read More
Git push requires username and password

I cloned a Git repository from my GitHub account to my PC. I want to work with both my PC and laptop, but with one GitHub account. When I try to push to or pull from GitHub using my PC, …

Read More
Git push results in "Authentication Failed"

I have been using Github for a little while and I have been fine with git add, git commit, and git push so far with no problems. Suddenly I am having an error that says: fatal: Authentication Failed In the …

Read More