Accessing Apache Kafka on OpenShift using its REST API



In this tutorial we will learn how access an Apache Kafka cluster running on OpenShift / Kubernetes using the Kafka Bridge.

The Apache Kafka Bridge includes a RESTful interface that let HTTP-based clients to interact with a Kafka cluster. This can simplify accessing the Kafka cluster which might be running on a Cloud Environment such as Kubernates or OpenShift.

In order to complete this tutorial, install the Strimzi Operator, the Kafka cluster and a Topic as discussed in this tutorial: Apache Kafka on OpenShift quickstart guide

Now that the Strimzi Operator has been installed, let's create a Kafka Bridge. Using your editor, create this yaml file (available at: https://github.com/fmarchioni/masteringintegration/blob/master/kafka/openshift/kafka-bridge.yaml)

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaBridge
metadata:
  name: my-bridge
  namespace: kafka-demo
spec:
  replicas: 1
  bootstrapServers: 'my-cluster-kafka-bootstrap:9092'
  http:
    port: 8080

This file will create a Kafka Bridge which points to the Kafka cluster via the bootstrap servers available at: 'my-cluster-kafka-bootstrap:9092'

oc create -f kafka-bridge.yaml

Verify that the service has been created:

oc get services
NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
my-bridge-bridge-service    ClusterIP   172.30.43.5    <none>        8080/TCP                     12s
my-kafka-kafka-bootstrap    ClusterIP   172.30.4.52    <none>        9091/TCP,9092/TCP,9093/TCP   3h2m
my-kafka-kafka-brokers      ClusterIP   None           <none>        9091/TCP,9092/TCP,9093/TCP   3h2m
my-kafka-zookeeper-client   ClusterIP   172.30.39.12   <none>        2181/TCP                     3h3m
my-kafka-zookeeper-nodes    ClusterIP   None           <none>        2181/TCP,2888/TCP,3888/TCP   3h3m

Next, expose the service through a Route:

oc expose service my-bridge-bridge-service

Check that the Route is available:

oc get routes
NAME                       HOST/PORT                                              PATH   SERVICES                   PORT       TERMINATION   WILDCARD
my-bridge-bridge-service   my-bridge-bridge-service-kafka-demo.apps-crc.testing          my-bridge-bridge-service   rest-api    

Create a Kafka Producer

In order to create a Kafka producer using the REST API, we will send a JSON formatted POST to the Router address which is available outside of OpenShift:

curl -X POST \
  http://my-bridge-bridge-service-kafka-demo.apps-crc.testing/topics/demo-queue \
  -H 'content-type: application/vnd.kafka.json.v2+json' \
  -d '{
    "records": [
        {
            "key": "my-key1",
            "value": "value-0001"
        },
        {
            "key": "my-key2",
            "value": "value-0002"
        }
    ]
}'

Provided that you have created the Topic "demo-queue", the above command will complete successfully.

Creating a Kafka Consumer

To create a Kafka Consumer using the REST API is a three step procedure. At first, you need to define a Consumer Group:

curl -X POST http://my-bridge-bridge-service-kafka-demo.apps-crc.testing/consumers/bridge-quickstart-consumer-group \
  -H 'content-type: application/vnd.kafka.v2+json' \
  -d '{
    "name": "bridge-quickstart-consumer",
    "auto.offset.reset": "earliest",
    "format": "json",
    "enable.auto.commit": false,
    "fetch.min.bytes": 512,
    "consumer.request.timeout.ms": 30000
  }'

Next, attach the Consumer Group to a Topic, for example the 'demo-queue':

 curl -X POST http://my-bridge-bridge-service-kafka-demo.apps-crc.testing/consumers/bridge-quickstart-consumer-group/instances/bridge-quickstart-consumer/subscription \
  -H 'content-type: application/vnd.kafka.v2+json' \
  -d '{
    "topics": [
        "demo-queue"
    ]
}'

Finally, you can retrieve messages by sending a GET to the Consumer Group which we have defined:

curl -X GET http://my-bridge-bridge-service-kafka-demo.apps-crc.testing/consumers/bridge-quickstart-consumer-group/instances/bridge-quickstart-consumer/records \
  -H 'accept: application/vnd.kafka.json.v2+json'

After creating and subscribing to a Kafka Bridge consumer, a first GET request will return an empty response because the poll operation starts a rebalancing process to assign partitions.

Repeat step two to retrieve messages from the Kafka Bridge consumer.

[{"topic":"demo-queue","key":"my-key1","value":"value-0001","partition":0,"offset":8},{"topic":"demo-queue","key":"my-key2","value":"value-0002","partition":0,"offset":9}]

Congratulations! You have managed to connect to a Kafka cluster on OpenShift using the Bridge API. For more details on the Bridge REST API check https://strimzi.io/docs/bridge/latest/

FREE WildFly Application Server - JBoss - Quarkus - Drools Tutorials
Cookie Policy

Cookie Policy This website uses cookies that are necessary to its functioning and required to achieve the purposes illustrated in the privacy policy. By accepting this OR scrolling this page OR continuing to browse, you agree to our privacy policy.

© 2020 Your Company. All Rights Reserved. Designed By JoomShaper

Please publish modules in offcanvas position.