Setting up a Mutiple Broker Kafka Cluster

In this tutorial we will learn how to start a Single Node with Mutiple Broker Kafka Cluster and produce/consume messages with it.

The architecture of our Kafka cluster will look like this picture:

In this tutorial we will learn how to start a Single Node with Mutiple Broker Kafka Cluster and produce/consume messages with it.

First, start the ZooKeeper server. Kafka provides a simple ZooKeeper configuration file to launch a single ZooKeeper instance. To install the ZooKeeper instance, use the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties  [2020-04-14 19:42:12,456] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2020-04-14 19:42:12,470] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase) [2020-04-14 19:42:12,475] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2020-04-14 19:42:12,480] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2020-04-14 19:42:12,496] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager) 

Starting the Kafka Broker

After start ZooKeeper , we need to create a distinct property file for each Broker. Copy the original file config/server.properties so to make a copy for each server:

cp config/server.properties config/server1.properties cp config/server.properties config/server2.properties 

Now, edit each file and include distinct settings for the following properties.

Server1:

broker.id=1 port=9093 log.dir=/tmp/kafka-logs-1 advertised.listeners=PLAINTEXT://127.0.0.1:9093 

Server2:

broker.id=2 port=9094 log.dir=/tmp/kafka-logs-2 advertised.listeners=PLAINTEXT://127.0.0.1:9094 

Now start the two Kafka brokers with the following commands:

bin/kafka-server-start.sh config/server1.properties  [2020-04-15 13:04:12,342] INFO [SocketServer brokerId=1] Started data-plane processors for 1 acceptors (kafka.network.SocketServer) [2020-04-15 13:04:12,348] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:04:12,348] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:04:12,348] INFO Kafka startTimeMs: 1586948652342 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:04:12,352] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)  

And the second one:

bin/kafka-server-start.sh config/server2.properties  [2020-04-15 13:06:03,786] INFO [SocketServer brokerId=2] Started data-plane processors for 1 acceptors (kafka.network.SocketServer) [2020-04-15 13:06:03,790] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:06:03,790] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:06:03,790] INFO Kafka startTimeMs: 1586948763787 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:06:03,792] INFO [KafkaServer id=2] started (kafka.server.KafkaServer) 

Creating a Topic

Kafka has the create command to create topics. Let’s create a topic called my-topic:

 bin/kafka-topics.sh --create --bootstrap-server localhost:9093 --replication-factor 2 --partitions 1 --topic my-topic  

Here:

  • replication-factor 2 indicates up to two replicas
  • partition 1 indicates one partition
 bin/kafka-topics.sh --describe --bootstrap-server localhost:9093 --topic my-topic Topic: my-topic	PartitionCount: 1	ReplicationFactor: 2	Configs: segment.bytes=1073741824 	Topic: my-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2 

Starting a Producer

You can use the kafka-console-producer shell script to start producers . It accepts input from the command line and publishes them as messages. By default, each new line is considered a message.

 bin/kafka-console-producer.sh --broker-list localhost:9093 --topic my-topic 

Now type the following:

Hello cluster! [Enter] 

Starting a Consumer

Kafka has the kafka-console-consumer script to start a message consumer client . It shows the output at the command line as soon as it subscribes to the topic in the broker:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9093 --from-beginning --topic my-topic 

you see the following output:

Found the article helpful? if so please follow us on Socials
Twitter Icon       Facebook Icon       LinkedIn Icon       Mastodon Icon