In this tutorial we will learn how to get started quickly with Apache Kafka on OpenShift. The same concepts apply if you want to run Apache Kafka on any Kubernates compliant environment
In order to kickstart Apache Kafka on OpenShift, we will install the Strimzi Operator. The project Strimzi (https://strimzi.io/) simplifies the process of running Apache Kafka in a Kubernetes cluster in various deployment configurations.
Let’s create a new OpenShift project named ‘kafka-demo’
oc new-project kafka-demo The simplest way to install the Strimzi Operator is to use directly the Web console of OpenShift. On the left panel, choose Operators | Operator Hub. From there, search the “strimzi” Operator as depicted by this image:
Then, choose to install the Operator in your current namespace (kafka-demo):
In a couple of minutes or less, the Operator will be available in your Pod list:
oc get pods NAME READY STATUS RESTARTS AGE strimzi-cluster-operator-v0.18.0-587d76b4b9-vbfqq 1/1 Running 0 77s Then, we can create a Kafka cluster using a resource whose kind is “Kafka”. For this purpose we can use this yaml file (you can download it from here: https://github.com/fmarchioni/masteringintegration/blob/master/kafka/openshift/kafka-cluster-descriptor.yaml):
apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata: name: my-kafka namespace: kafka-demo spec: kafka: version: 2.5.0 replicas: 3 listeners: plain: {} tls: {} config: offsets.topic.replication.factor: 3 transaction.state.log.replication.factor: 3 transaction.state.log.min.isr: 2 log.message.format.version: '2.3' storage: type: ephemeral zookeeper: replicas: 3 storage: type: ephemeral entityOperator: topicOperator: {} userOperator: {}
Simply create the cluster from this resource:
oc create -f kafka-cluster-descriptor.yaml kafka.kafka.strimzi.io/my-kafka created Then, wait that all Kafka and Zookeper Pods are up and running:
oc get pods NAME READY STATUS RESTARTS AGE my-kafka-zookeeper-0 1/1 Running 0 27s my-kafka-zookeeper-1 1/1 Running 0 27s my-kafka-zookeeper-2 1/1 Running 0 27s strimzi-cluster-operator-v0.18.0-587d76b4b9-vbfqq 1/1 Running 0 2m18s Finally, we will be creating a Topic to test our cluster. The following yaml file (available here: https://github.com/fmarchioni/masteringintegration/blob/master/kafka/openshift/kafka-topic-queue-descriptor.yaml) will create a Topic named “demo-queue”:
apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaTopic metadata: name: demo-queue labels: strimzi.io/cluster: my-kafka namespace: kafka-demo spec: partitions: 1 replicas: 1 config: retention.ms: 604800000 segment.bytes: 1073741824
Create the resource as follows:
oc create -f kafka-topic-queue-descriptor.yaml kafkatopic.kafka.strimzi.io/demo-queue created Now let’s try the cluster by logging into one of our Kafka brokers:
oc get pods NAME READY STATUS RESTARTS AGE my-kafka-entity-operator-5948566694-php8v 3/3 Running 0 3m50s my-kafka-kafka-0 2/2 Running 0 4m18s my-kafka-kafka-1 2/2 Running 0 4m18s my-kafka-kafka-2 2/2 Running 0 4m18s my-kafka-zookeeper-0 1/1 Running 0 4m47s my-kafka-zookeeper-1 1/1 Running 0 4m47s my-kafka-zookeeper-2 1/1 Running 0 4m47s We will be logging into the Pod named my-kafka-kafka-0:
$ oc rsh my-kafka-kafka-0 And once logged, start the kafka-console-producer.sh pointing to the demo-queue:
/opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic demo-queue From another shell, log again into the Pod and start the kafka-console-consumer.sh as follows:
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo-queue --from-beginning Then start typing some text to produce messages:
And here’s the Consumer:
Congratulations! You have managed to install a Kafka cluster on OpenShift using the Strimzi Operator and managed to send test messages across the cluster!
Found the article helpful? if so please follow us on Socials
