Getting started with Apache Kafka



In this tutorial we will learn how to start a Single Node with Single Broker Kafka Cluster and produce/consume messages with it.

In Kafka, there are three types of clusters:

• Single node–single broker

• Single node–multiple broker

• Multiple node–multiple broker

An example diagram of a single node– single broker cluster is shown in the following picture:

kafka tutorial kafka tutorial kafka tutorial

At first download and unzip Apache Kafka from https://kafka.apache.org/downloads

Starting the Zookeeper Server

Next, start the ZooKeeper server. Kafka provides a simple ZooKeeper configuration file to launch a single ZooKeeper instance. To install the ZooKeeper instance, use the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties
 
[2020-04-14 19:42:12,456] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-14 19:42:12,470] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2020-04-14 19:42:12,475] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-14 19:42:12,480] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-14 19:42:12,496] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)

The following are the main properties defined in zookeeper.properties:

  • dataDir . The data directory where ZooKeeper is stored (default /tmp/zookeeper)
  • clientPort . The listening port for client requests. By default, ZooKeeper listens in the 2181 TCP port
  • maxClientCnxns . The limit per IP for the number of connections (0 = unbounded):

Starting the Kafka Broker

You can start the Kafka Broker using the default configuration (config.properties)

NOTE: If you are using localhost as bind address, we recommend setting the advertised.listeners property as follows otherwise, it will use the valuereturned from java.net.InetAddress.getCanonicalHostName():

advertised.listeners=PLAINTEXT://localhost:9092

Now , start the Kafka broker with the following command:

bin/kafka-server-start.sh config/server.properties

[2020-04-14 19:43:13,350] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-14 19:43:13,350] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-14 19:43:13,351] INFO Kafka startTimeMs: 1586886193347 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-14 19:43:13,352] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

The following are the main properties defined in server.properties:

  • Broker id: The unique positive integer id for each broker.
  • Port: The port where the socket server listens on (default port=9092).
  • Log dir: The directory to store log files (default log.dir=/tmp/kafka10-logs).
  • Num partitions: The number of log partitions per topic (default num.partitions=2).
  • ZooKeeper connect: The ZooKeeper connection URL (default zookeeper.connect=localhost:2181)

Creating a Topic

Kafka has the create command to create topics. Let’s create a topic called myTopic with one partition and one replica:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic myTopic

To obtain the list of topics on any Kafka server, use the following command:

bin/kafka-topics.sh --list --zookeeper localhost:2181
myTopic

Starting a Producer

You can use the kafka-console-producer shell script to start producers . It accepts input from the command line and publishes them as messages. By default, each new line is considered a message.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic myTopic

Now type the following:

Hello World! [Enter]

Single Node–Single Broker Cluster

Starting a Consumer

Kafka has the kafka-console-consumer script to start a message consumer client . It shows the output at the command line as soon as it subscribes to the topic in the broker:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTopic --from-beginning

you see the following output:

Hello World!

Single Node–Single Broker Cluster

FREE WildFly Application Server - JBoss - Quarkus - Drools Tutorials
Cookie Policy

Cookie Policy This website uses cookies that are necessary to its functioning and required to achieve the purposes illustrated in the privacy policy. By accepting this OR scrolling this page OR continuing to browse, you agree to our privacy policy.

© 2020 Your Company. All Rights Reserved. Designed By JoomShaper

Please publish modules in offcanvas position.