Setting up a Mutiple Broker Kafka Cluster



In this tutorial we will learn how to start a Single Node with Mutiple Broker Kafka Cluster and produce/consume messages with it.

The architecture of our Kafka cluster will look like this picture:

In this tutorial we will learn how to start a Single Node with Mutiple Broker Kafka Cluster and produce/consume messages with it.

First, start the ZooKeeper server. Kafka provides a simple ZooKeeper configuration file to launch a single ZooKeeper instance. To install the ZooKeeper instance, use the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties

[2020-04-14 19:42:12,456] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-14 19:42:12,470] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2020-04-14 19:42:12,475] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-14 19:42:12,480] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-14 19:42:12,496] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)

Starting the Kafka Broker

After start ZooKeeper , we need to create a distinct property file for each Broker. Copy the original file config/server.properties so to make a copy for each server:

cp config/server.properties config/server1.properties
cp config/server.properties config/server2.properties

Now, edit each file and include distinct settings for the following properties.

Server1:

broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1
advertised.listeners=PLAINTEXT://127.0.0.1:9093

Server2:

broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2
advertised.listeners=PLAINTEXT://127.0.0.1:9094

Now start the two Kafka brokers with the following commands:

bin/kafka-server-start.sh config/server1.properties

[2020-04-15 13:04:12,342] INFO [SocketServer brokerId=1] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
[2020-04-15 13:04:12,348] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:04:12,348] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:04:12,348] INFO Kafka startTimeMs: 1586948652342 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:04:12,352] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

And the second one:

bin/kafka-server-start.sh config/server2.properties

[2020-04-15 13:06:03,786] INFO [SocketServer brokerId=2] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
[2020-04-15 13:06:03,790] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:06:03,790] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:06:03,790] INFO Kafka startTimeMs: 1586948763787 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:06:03,792] INFO [KafkaServer id=2] started (kafka.server.KafkaServer)

Creating a Topic

Kafka has the create command to create topics. Let’s create a topic called my-topic:

 bin/kafka-topics.sh --create --bootstrap-server localhost:9093 --replication-factor 2 --partitions 1 --topic my-topic

Here:

  • replication-factor 2 indicates up to two replicas
  • partition 1 indicates one partition
 bin/kafka-topics.sh --describe --bootstrap-server localhost:9093 --topic my-topic
Topic: my-topic	PartitionCount: 1	ReplicationFactor: 2	Configs: segment.bytes=1073741824
	Topic: my-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

Starting a Producer

You can use the kafka-console-producer shell script to start producers . It accepts input from the command line and publishes them as messages. By default, each new line is considered a message.

 bin/kafka-console-producer.sh --broker-list localhost:9093 --topic my-topic

Now type the following:

Hello cluster! [Enter]

Starting a Consumer

Kafka has the kafka-console-consumer script to start a message consumer client . It shows the output at the command line as soon as it subscribes to the topic in the broker:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9093 --from-beginning --topic my-topic

you see the following output:

FREE WildFly Application Server - JBoss - Quarkus - Drools Tutorials
Cookie Policy

Cookie Policy This website uses cookies that are necessary to its functioning and required to achieve the purposes illustrated in the privacy policy. By accepting this OR scrolling this page OR continuing to browse, you agree to our privacy policy.

© 2020 Your Company. All Rights Reserved. Designed By JoomShaper

Please publish modules in offcanvas position.