Setting up a Mutiple Broker Kafka Cluster

In this tutorial we will learn how to start a Single Node with Mutiple Broker Kafka Cluster and produce/consume messages with it.

The architecture of our Kafka cluster will look like this picture:

In this tutorial we will learn how to start a Single Node with Mutiple Broker Kafka Cluster and produce/consume messages with it.

First, start the ZooKeeper server. Kafka provides a simple ZooKeeper configuration file to launch a single ZooKeeper instance. To install the ZooKeeper instance, use the following command:

bin/ config/  [2020-04-14 19:42:12,456] INFO binding to port (org.apache.zookeeper.server.NIOServerCnxnFactory) [2020-04-14 19:42:12,470] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase) [2020-04-14 19:42:12,475] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2020-04-14 19:42:12,480] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2020-04-14 19:42:12,496] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager) 

Starting the Kafka Broker

After start ZooKeeper , we need to create a distinct property file for each Broker. Copy the original file config/ so to make a copy for each server:

cp config/ config/ cp config/ config/ 

Now, edit each file and include distinct settings for the following properties.

Server1: port=9093 log.dir=/tmp/kafka-logs-1 advertised.listeners=PLAINTEXT:// 

Server2: port=9094 log.dir=/tmp/kafka-logs-2 advertised.listeners=PLAINTEXT:// 

Now start the two Kafka brokers with the following commands:

bin/ config/  [2020-04-15 13:04:12,342] INFO [SocketServer brokerId=1] Started data-plane processors for 1 acceptors ( [2020-04-15 13:04:12,348] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:04:12,348] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:04:12,348] INFO Kafka startTimeMs: 1586948652342 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:04:12,352] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)  

And the second one:

bin/ config/  [2020-04-15 13:06:03,786] INFO [SocketServer brokerId=2] Started data-plane processors for 1 acceptors ( [2020-04-15 13:06:03,790] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:06:03,790] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:06:03,790] INFO Kafka startTimeMs: 1586948763787 (org.apache.kafka.common.utils.AppInfoParser) [2020-04-15 13:06:03,792] INFO [KafkaServer id=2] started (kafka.server.KafkaServer) 

Creating a Topic

Kafka has the create command to create topics. Let’s create a topic called my-topic:

 bin/ --create --bootstrap-server localhost:9093 --replication-factor 2 --partitions 1 --topic my-topic  


  • replication-factor 2 indicates up to two replicas
  • partition 1 indicates one partition
 bin/ --describe --bootstrap-server localhost:9093 --topic my-topic Topic: my-topic	PartitionCount: 1	ReplicationFactor: 2	Configs: segment.bytes=1073741824 	Topic: my-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2 

Starting a Producer

You can use the kafka-console-producer shell script to start producers . It accepts input from the command line and publishes them as messages. By default, each new line is considered a message.

 bin/ --broker-list localhost:9093 --topic my-topic 

Now type the following:

Hello cluster! [Enter] 

Starting a Consumer

Kafka has the kafka-console-consumer script to start a message consumer client . It shows the output at the command line as soon as it subscribes to the topic in the broker:

bin/ --bootstrap-server localhost:9093 --from-beginning --topic my-topic 

you see the following output: