How to publish messages directly from existing Mainframe application to Kafka topics?

apache-kafka cobol mainframe zos
Sunny Goel picture Sunny Goel · Aug 6, 2017 · Viewed 7.6k times · Source

We are planning to replace IBM MQ with Apache Kafka to implement pub-sub use-cases in future. Currently Mainframe(running Z/OS) applications are written using COBOL programming language & pushing messages to MQ. I went through the documentation to find out about Clients supported by Kafka and seems, it doesn't support COBOL.

What's the best approach to build producer on Mainframe and publish messages directly to Kafka topics with minimal change in existing applications ?

Answer

Hans Jespersen picture Hans Jespersen · Aug 6, 2017

Lots of options. You can publish to Kafka using any REST/HTTP client in any language. You can leave WebSphere MQ on zOS and run an external Kafka Source Connector to pull from MQ and put into Kafka. You can write to DB2 or another database and use a number of CDC tools (including IBM InfoSphere) to send the database updates to Kafka. You can use the Confluent JMS client to publish using Java. You can help finish the work to get librdkafka C client to run native on zOS. You could use Datapower as an ESB gateway between your mainframe app and Kafka via the Confluent REST Proxy. It all depends on your use case which option is best.

Some Links of interests:

Confluent Kafka JMS Client http://docs.confluent.io/current/clients/kafka-jms-client/docs/index.html

DataMountaineer Kafka Connectors for JMS Source: http://docs.datamountaineer.com/en/latest/jms-source.html

Sink: http://docs.datamountaineer.com/en/latest/jms.html

Related questions

List all kafka topics

I'm using kafka 0.10 without zookeeper. I want to get kafka topics list. This command is not working since we're not using zookeeper: bin/kafka-topics.sh --list --zookeeper localhost:2181. How can I get the same output without zookeeper?

Read More
Purge Kafka Topic

Is there a way to purge the topic in kafka? I pushed a message that was too big into a kafka message topic on my local machine, now I'm getting an error: kafka.common.InvalidMessageSizeException: invalid message size Increasing the …

Read More
Is there a way to delete all the data from a topic or delete the topic before every run?

Is there a way to delete all the data from a topic or delete the topic before every run? Can I modify the KafkaConfig.scala file to change the logRetentionHours property? Is there a way the messages gets deleted as …

Read More