Apache Kafka on OpenShift quickstart guide



In this tutorial we will learn how to get started quickly with Apache Kafka on OpenShift. The same concepts apply if you want to run Apache Kafka on any Kubernates compliant environment

In order to kickstart Apache Kafka on OpenShift, we will install the Strimzi Operator. The project Strimzi (https://strimzi.io/) simplifies the process of running Apache Kafka in a Kubernetes cluster in various deployment configurations.

Let's create a new OpenShift project named 'kafka-demo'

oc new-project kafka-demo

The simplest way to install the Strimzi Operator is to use directly the Web console of OpenShift. On the left panel, choose Operators | Operator Hub. From there, search the "strimzi" Operator as depicted by this image:

Apache Kafka on OpenShift tutorial Openshift

Then, choose to install the Operator in your current namespace (kafka-demo):

Apache Kafka on OpenShift tutorial Openshift

In a couple of minutes or less, the Operator will be available in your Pod list:

oc get pods
NAME                                                READY   STATUS    RESTARTS   AGE
strimzi-cluster-operator-v0.18.0-587d76b4b9-vbfqq   1/1     Running   0          77s

Then, we can create a Kafka cluster using a resource whose kind is "Kafka". For this purpose we can use this yaml file (you can download it from here: https://github.com/fmarchioni/masteringintegration/blob/master/kafka/openshift/kafka-cluster-descriptor.yaml):

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-kafka
  namespace: kafka-demo
spec:
  kafka:
    version: 2.5.0
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: '2.3'
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

Simply create the cluster from this resource:

oc create -f kafka-cluster-descriptor.yaml 
kafka.kafka.strimzi.io/my-kafka created

Then, wait that all Kafka and Zookeper Pods are up and running:

oc get pods
NAME                                                READY   STATUS    RESTARTS   AGE
my-kafka-zookeeper-0                                1/1     Running   0          27s
my-kafka-zookeeper-1                                1/1     Running   0          27s
my-kafka-zookeeper-2                                1/1     Running   0          27s
strimzi-cluster-operator-v0.18.0-587d76b4b9-vbfqq   1/1     Running   0          2m18s

Finally, we will be creating a Topic to test our cluster. The following yaml file (available here: https://github.com/fmarchioni/masteringintegration/blob/master/kafka/openshift/kafka-topic-queue-descriptor.yaml) will create a Topic named "demo-queue":

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: demo-queue
  labels:
    strimzi.io/cluster: my-kafka
  namespace: kafka-demo
spec:
  partitions: 1
  replicas: 1
  config:
    retention.ms: 604800000
    segment.bytes: 1073741824

Create the resource as follows:

oc create -f kafka-topic-queue-descriptor.yaml 

kafkatopic.kafka.strimzi.io/demo-queue created

Now let's try the cluster by logging into one of our Kafka brokers:

oc get pods
NAME                                                READY   STATUS    RESTARTS   AGE
my-kafka-entity-operator-5948566694-php8v           3/3     Running   0          3m50s
my-kafka-kafka-0                                    2/2     Running   0          4m18s
my-kafka-kafka-1                                    2/2     Running   0          4m18s
my-kafka-kafka-2                                    2/2     Running   0          4m18s
my-kafka-zookeeper-0                                1/1     Running   0          4m47s
my-kafka-zookeeper-1                                1/1     Running   0          4m47s
my-kafka-zookeeper-2                                1/1     Running   0          4m47s

We will be logging into the Pod named my-kafka-kafka-0:

$ oc rsh my-kafka-kafka-0

And once logged, start the kafka-console-producer.sh pointing to the demo-queue:

 /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic demo-queue

From another shell, log again into the Pod and start the kafka-console-consumer.sh as follows:

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo-queue --from-beginning

Then start typing some text to produce messages:

Apache Kafka on OpenShift tutorial Openshift

And here's the Consumer:

Apache Kafka on OpenShift tutorial Openshift

Congratulations! You have managed to install a Kafka cluster on OpenShift using the Strimzi Operator and managed to send test messages across the cluster!

FREE WildFly Application Server - JBoss - Quarkus - Drools Tutorials
Cookie Policy

Cookie Policy This website uses cookies that are necessary to its functioning and required to achieve the purposes illustrated in the privacy policy. By accepting this OR scrolling this page OR continuing to browse, you agree to our privacy policy.

© 2020 Your Company. All Rights Reserved. Designed By JoomShaper

Please publish modules in offcanvas position.