Apache Kafka on OpenShift quickstart guide

In this tutorial we will learn how to get started quickly with Apache Kafka on OpenShift. The same concepts apply if you want to run Apache Kafka on any Kubernates compliant environment

In order to kickstart Apache Kafka on OpenShift, we will install the Strimzi Operator. The project Strimzi (https://strimzi.io/) simplifies the process of running Apache Kafka in a Kubernetes cluster in various deployment configurations.

Let’s create a new OpenShift project named ‘kafka-demo’

oc new-project kafka-demo 

The simplest way to install the Strimzi Operator is to use directly the Web console of OpenShift. On the left panel, choose Operators | Operator Hub. From there, search the “strimzi” Operator as depicted by this image:

Apache Kafka on OpenShift tutorial Openshift

Then, choose to install the Operator in your current namespace (kafka-demo):

Apache Kafka on OpenShift tutorial Openshift

In a couple of minutes or less, the Operator will be available in your Pod list:

oc get pods NAME                                                READY   STATUS    RESTARTS   AGE strimzi-cluster-operator-v0.18.0-587d76b4b9-vbfqq   1/1     Running   0          77s 

Then, we can create a Kafka cluster using a resource whose kind is “Kafka”. For this purpose we can use this yaml file (you can download it from here: https://github.com/fmarchioni/masteringintegration/blob/master/kafka/openshift/kafka-cluster-descriptor.yaml):

apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata:   name: my-kafka   namespace: kafka-demo spec:   kafka:     version: 2.5.0     replicas: 3     listeners:       plain: {}       tls: {}     config:       offsets.topic.replication.factor: 3       transaction.state.log.replication.factor: 3       transaction.state.log.min.isr: 2       log.message.format.version: '2.3'     storage:       type: ephemeral   zookeeper:     replicas: 3     storage:       type: ephemeral   entityOperator:     topicOperator: {}     userOperator: {} 

Simply create the cluster from this resource:

oc create -f kafka-cluster-descriptor.yaml  kafka.kafka.strimzi.io/my-kafka created 

Then, wait that all Kafka and Zookeper Pods are up and running:

oc get pods NAME                                                READY   STATUS    RESTARTS   AGE my-kafka-zookeeper-0                                1/1     Running   0          27s my-kafka-zookeeper-1                                1/1     Running   0          27s my-kafka-zookeeper-2                                1/1     Running   0          27s strimzi-cluster-operator-v0.18.0-587d76b4b9-vbfqq   1/1     Running   0          2m18s 

Finally, we will be creating a Topic to test our cluster. The following yaml file (available here: https://github.com/fmarchioni/masteringintegration/blob/master/kafka/openshift/kafka-topic-queue-descriptor.yaml) will create a Topic named “demo-queue”:

apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaTopic metadata:   name: demo-queue   labels:     strimzi.io/cluster: my-kafka   namespace: kafka-demo spec:   partitions: 1   replicas: 1   config:     retention.ms: 604800000     segment.bytes: 1073741824 

Create the resource as follows:

oc create -f kafka-topic-queue-descriptor.yaml   kafkatopic.kafka.strimzi.io/demo-queue created 

Now let’s try the cluster by logging into one of our Kafka brokers:

oc get pods NAME                                                READY   STATUS    RESTARTS   AGE my-kafka-entity-operator-5948566694-php8v           3/3     Running   0          3m50s my-kafka-kafka-0                                    2/2     Running   0          4m18s my-kafka-kafka-1                                    2/2     Running   0          4m18s my-kafka-kafka-2                                    2/2     Running   0          4m18s my-kafka-zookeeper-0                                1/1     Running   0          4m47s my-kafka-zookeeper-1                                1/1     Running   0          4m47s my-kafka-zookeeper-2                                1/1     Running   0          4m47s 

We will be logging into the Pod named my-kafka-kafka-0:

$ oc rsh my-kafka-kafka-0 

And once logged, start the kafka-console-producer.sh pointing to the demo-queue:

 /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic demo-queue 

From another shell, log again into the Pod and start the kafka-console-consumer.sh as follows:

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo-queue --from-beginning 

Then start typing some text to produce messages:

Apache Kafka on OpenShift tutorial Openshift

And here’s the Consumer:

Apache Kafka on OpenShift tutorial Openshift

Congratulations! You have managed to install a Kafka cluster on OpenShift using the Strimzi Operator and managed to send test messages across the cluster!