Kafka deployment on minikube

Question

Hi i am new to kubernets, i am using minikube single node cluster for local development and testing.

Host: Ubuntu 16.04 LTS.
Minikube: Virtual box running minikube cluster

My requirement is i need to deploy kafka and zookeeper on minikube and should be used to produce or consume messages. I followed this link and successfully deployed it on minikube its details are below

$ kubectl get services
NAME            CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
kafka-service   10.0.0.15    <pending>     9092:30244/TCP               46m
kubernetes      10.0.0.1     <none>        443/TCP                      53m
zoo1            10.0.0.43    <none>        2181/TCP,2888/TCP,3888/TCP   50m
zoo2            10.0.0.226   <none>        2181/TCP,2888/TCP,3888/TCP   50m
zoo3            10.0.0.6     <none>        2181/TCP,2888/TCP,3888/TCP   50m

$ kubectl get pods
NAME                                      READY     STATUS    RESTARTS   AGE
kafka-deployment-3583985961-f2301         1/1       Running   0          48m
zookeeper-deployment-1-1598963595-vgx1l   1/1       Running   0          52m
zookeeper-deployment-2-2038841231-tdsff   1/1       Running   0          52m
zookeeper-deployment-3-2478718867-5vjcj   1/1       Running   0          52m

$ kubectl describe service kafka-service
Name:           kafka-service
Namespace:      default
Labels:         app=kafka
Annotations:    <none>
Selector:       app=kafka
Type:           LoadBalancer
IP:             10.0.0.15
Port:           kafka-port  9092/TCP
NodePort:       kafka-port  30244/TCP
Endpoints:      172.17.0.7:9092
Session Affinity:   None
Events:         <none>

and i have set KAFKA_ADVERTISED_HOST_NAME to minikube ip(192.168.99.100). Now for message producer i am using $cat textfile.log | kafkacat -b $(minikube ip):30244 -t mytopic its not publishing the message giving below message

% Auto-selecting Producer mode (use -P or -C to override)
% Delivery failed for message: Local: Message timed out

can any one help how to publish and consume message.

Answer 1

I know that this is quite an old post. Were you able to resolve and run kafka + zookeeper within minikube? I was able to run a simple single cluster kafka and zookeeper deployment successfully using minikube v0.17.1 and produce and consume messages using kafkacat producer and consumer respectively. I was able to run these successfully on uBuntu and Mac OSX. The deployment and service yamls are as below:

zookeeper-deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: zookeeper
  name: zookeeper
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - image: wurstmeister/zookeeper
        imagePullPolicy: IfNotPresent
        name: zookeeper
        ports:
        - containerPort: 2181

zookeeper-service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: zookeeper-service
  name: zookeeper-service
spec:
  type: NodePort
  ports:
  - name: zookeeper-port
    port: 2181
    nodePort: 30181
    targetPort: 2181
  selector:
    app: zookeeper

kafka-deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: kafka
  name: kafka
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - env:
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: "192.168.99.100"
        - name: KAFKA_ADVERTISED_PORT
          value: "30092"
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: 192.168.99.100:30181
        - name: KAFKA_CREATE_TOPICS
          value: "test-topic:1:1"
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        name: kafka
        ports:
        - containerPort: 9092

kafka-service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-service
  name: kafka-service
spec:
  type: NodePort
  ports:
  - name: kafka-port
    port: 9092
    nodePort: 30092
    targetPort: 9092
  selector:
    app: kafka

You can test your deployment by installing kafkacat client and running the following commands on separate terminal windows:

echo "Am I receiving this message?" | kafkacat -P -b 192.168.99.100:30092 -t test-topic

kafkacat -C -b 192.168.99.100:30092 -t test-topic

% Reached end of topic test-topic [0] at offset 0

Am I receiving this message?

I was able to successfully run this on minikube versions v0.17.1 and v0.19.0. If you want to run this on minikube versions v0.21.1 and v0.23.0, please refer to my reply to the post here: Kafka inaccessible once inside Kubernetes/Minikube

Thanks.