Does Kafka python API support stream processing?

python apache-kafka apache-kafka-streams confluent-platform kafka-python
user3126637 picture user3126637 · Aug 19, 2018 · Viewed 12.5k times · Source

I have used Kafka Streams in Java. I could not find similar API in python. Do Apache Kafka support stream processing in python?

Answer

OneCricketeer picture OneCricketeer · Aug 19, 2018

Kafka Streams is only available as a JVM library, but there are at least two Python implementations of it

In theory, you could try playing with Jython or Py4j to support it the JVM implementation, but otherwise you're stuck with consumer/producer or invoking the KSQL REST interface.

Outside of those options, you can also try Apache Beam, Flink or Spark, but they each require an external cluster scheduler to scale out.

Related questions

How to get latest offset for a partition for a kafka topic?

I am using the Python high level consumer for Kafka and want to know the latest offsets for each partition of a topic. However I cannot get it to work. from kafka import TopicPartition from kafka.consumer import KafkaConsumer con = …

Read More
NoBrokersAvailable: NoBrokersAvailable-Kafka Error

i have already started to learn Kafka. Trying basic operations on it. I have stucked on a point which about the 'Brokers'. My kafka is running but when i want to create a partition. from kafka import TopicPartition (ERROR THERE) …

Read More
How to programmatically create a topic in Apache Kafka using Python

So far I haven't seen a python client that implements the creation of a topic explicitly without using the configuration option to create automatically the topics.

Read More