In this tutorial we will learn how to start a Single Node with Mutiple Broker Kafka Cluster and produce/consume messages with it.
The architecture of our Kafka cluster will look like this picture:
First, start the ZooKeeper server. Kafka provides a simple ZooKeeper configuration file to launch a single ZooKeeper instance. To install the ZooKeeper instance, use the following command:
bin/zookeeper-server-start.sh config/zookeeper.properties [2020-04-14 19:42:12,456] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2020-04-14 19:42:12,470] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase) [2020-04-14 19:42:12,475] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2020-04-14 19:42:12,480] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2020-04-14 19:42:12,496] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager) Starting the Kafka Broker
After start ZooKeeper , we need to create a distinct property file for each Broker. Copy the original file config/server.properties so to make a copy for each server:
cp config/server.properties config/server1.properties cp config/server.properties config/server2.properties Now, edit each file and include distinct settings for the following properties.
Server1:
broker.id=1 port=9093 log.dir=/tmp/kafka-logs-1 advertised.listeners=PLAINTEXT://127.0.0.1:9093
Server2:
broker.id=2 port=9094 log.dir=/tmp/kafka-logs-2 advertised.listeners=PLAINTEXT://127.0.0.1:9094
Now start the two Kafka brokers with the following commands:
bin/kafka-server-start.sh config/server1.properties [2020-04-15 13:04:12,342] INFO [SocketServer brokerId=1] Started data-plane processors for 1 acceptors (kafka.network.SocketServer) [2020-04-15 13:04:12,348] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:04:12,348] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:04:12,348] INFO Kafka startTimeMs: 1586948652342 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:04:12,352] INFO [KafkaServer id=1] started (kafka.server.KafkaServer) And the second one:
bin/kafka-server-start.sh config/server2.properties [2020-04-15 13:06:03,786] INFO [SocketServer brokerId=2] Started data-plane processors for 1 acceptors (kafka.network.SocketServer) [2020-04-15 13:06:03,790] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:06:03,790] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:06:03,790] INFO Kafka startTimeMs: 1586948763787 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:06:03,792] INFO [KafkaServer id=2] started (kafka.server.KafkaServer) Creating a Topic
Kafka has the create command to create topics. Let’s create a topic called my-topic:
bin/kafka-topics.sh --create --bootstrap-server localhost:9093 --replication-factor 2 --partitions 1 --topic my-topic Here:
- replication-factor 2 indicates up to two replicas
- partition 1 indicates one partition
bin/kafka-topics.sh --describe --bootstrap-server localhost:9093 --topic my-topic Topic: my-topic PartitionCount: 1 ReplicationFactor: 2 Configs: segment.bytes=1073741824 Topic: my-topic Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Starting a Producer
You can use the kafka-console-producer shell script to start producers . It accepts input from the command line and publishes them as messages. By default, each new line is considered a message.
bin/kafka-console-producer.sh --broker-list localhost:9093 --topic my-topic Now type the following:
Hello cluster! [Enter]
Starting a Consumer
Kafka has the kafka-console-consumer script to start a message consumer client . It shows the output at the command line as soon as it subscribes to the topic in the broker:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9093 --from-beginning --topic my-topic you see the following output:
Found the article helpful? if so please follow us on Socials
