This tutorial will teach you how to run Apache Kafka in KRaft mode which does not require a Zookeeper server to manage the cluster capablities. We will learn at first how does the Kafka KRaft mode work and then we will learn how to create a cluster without using Zookeeper.
Apache Kafka has officially deprecated ZooKeeper in version 3.5. The introduction of the KRaft mode feature not only enhances the scalability capabilities of Kafka but also simplifies the learning curve by eliminating the need to manage ZooKeeper. This development brings several benefits, making it easier for system administrators to monitor, administer, and support Kafka, while providing developers with a unified security model for the entire system. Additionally, it offers a lightweight single-process deployment option, making it easier to get started with Kafka.
So, how does Kafka KRaft mode work?
In the Kafka Raft Metadata mode, Kafka stores its metadata and configurations in an internal topic called “__cluster_metadata.” This topic consists of a single partition, and the active controller serves as its leader. Managed by an internal quorum, this internal topic is replicated across the cluster. As a result, the cluster nodes can now function as brokers, controllers, or even both (known as combined nodes).
When the KRaft mode is enabled, a few selected servers are designated as controllers, forming the internal quorum. These controllers can operate in either active or standby mode, ready to take over if the currently active controller fails or goes offline.
With KRaft mode, ZooKeeper nodes are replaced by KRaft Controllers. The process of starting up the controllers involves the following steps:
- Designation of Kafka Brokers as KRaft controllers.
- Election of a leader among the controllers.
- The leader controller contains all previously committed records.
- Kafka brokers fetch metadata from this KRaft leader.
In the KRaft mode, every Kafka server now includes an additional configuration parameter called “process.roles.” This parameter can be assigned the following values:
- “broker”: The Kafka server serves as a broker.
- “controller”: The Kafka server acts as a controller of the internal Raft quorum.
- “broker,controller”: The Kafka server performs both the roles of a controller and a broker.
In the next section of this tutorial we will see two basic examples of Apache Kafka in KRaft mode. A single node set up and then a cluster set up which we will run on the same machine,
Starting Apache Kafka in KRaft mode
In the first basic example, we will start a single Apache Kafka node in KRaft mode. Firstly, download the latest release of Apache Kafka and unzip it in a folder.
tar -xzf kafka_2.13-3.5.0.tgz
Then, generate a Random ID for the Kafka broker using the kafka-storage tool:
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
The command returns an ID that you will use to start your cluster. Then, set up log directories for each node in your Kafka cluster using the default configuration available in config/kraft/server.properties
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties
Finally, the Kafka broker.
bin/kafka-server-start.sh config/kraft/server.properties
Verify that the Broker has started correctly:
Starting a cluster of Apache Kafka in KRaft mode
In order to start multiple Nodes of Apache Kafka in KRaft mode, the first thing we need is creating a copy of our configurations so that we will use a configuration for each Node.
cp config/kraft/server.properties config/kraft/server1.properties cp config/kraft/server.properties config/kraft/server2.properties cp config/kraft/server.properties config/kraft/server3.properties
Now we have three configuration files: server1.properties, server2.properties and server3.properties.
Then, we need to edit each configuration file to provide an unique node.id, a set of listener addresses, the directory for logs and the Controllers host and port we need as Quorum for the cluster.
Therefore, for server1.properties we will change the following settings:
node.id=1 controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094 listeners=PLAINTEXT://:9092,CONTROLLER://:19092 advertised.listeners=PLAINTEXT://localhost:9092 log.dirs=/tmp/server1/kraft-combined-logs
Then, for server2.properties we will configure:
node.id=2 controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094 listeners=PLAINTEXT://:9093,CONTROLLER://:19093 advertised.listeners=PLAINTEXT://localhost:9093 log.dirs=/tmp/server2/kraft-combined-logs
Finally, for server3.properties configure the following attributes:
node.id=3 controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094 listeners=PLAINTEXT://:9094,CONTROLLER://:19094 advertised.listeners=PLAINTEXT://localhost:9094 log.dirs=/tmp/server3/kraft-combined-logs
Setting up Log directories
We have completed the configuration part. Before starting the cluster we need to set up the Log directories. As we did in our earlier example, start by gathering a Random Id:
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" $echo $KAFKA_CLUSTER_ID MkBMxmVvQuax6o7cm1NjZw
Then, set up Log directories for Server1:
./bin/kafka-storage.sh format -t MkBMxmVvQuax6o7cm1NjZw -c ./config/kraft/server1.properties
For Server2:
./bin/kafka-storage.sh format -t MkBMxmVvQuax6o7cm1NjZw -c ./config/kraft/server2.properties
And finally, for Server3:
./bin/kafka-storage.sh format -t MkBMxmVvQuax6o7cm1NjZw -c ./config/kraft/server3.properties
Starting the Kafka Cluster
Finally, start each Server of the Cluster.
Server 1:
./bin/kafka-server-start.sh ./config/kraft/server1.properties
Server2:
./bin/kafka-server-start.sh ./config/kraft/server2.properties
Server3:
./bin/kafka-server-start.sh ./config/kraft/server3.properties
Testing the Cluster
When all cluster nodes are up and running we will run a simple topic consumer and producer. Firstly, let’s create a new topic “topictest”
./bin/kafka-topics.sh --create --topic topictest --partitions 3 --replication-factor 3 --bootstrap-server localhost:9092
Then, verify the details about the Partition Count for your Topic:
bin/kafka-topics.sh --bootstrap-server localhost:9093 --describe --topic topictest Topic: topictest TopicId: pFkvKZJBQ5-9D2j97f7uAA PartitionCount: 3 ReplicationFactor: 3Configs: segment.bytes=1073741824 Topic: topictest Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 Topic: topictest Partition: 1 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 Topic: topictest Partition: 2 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
Finally, start from the terminal a Message producer:
bin/kafka-console-producer.sh --topic topictest --bootstrap-server localhost:9092
Then, from another terminal create a Message Consumer:
bin/kafka-console-consumer.sh --topic topictest --from-beginning --bootstrap-server localhost:9092
Verify that each Message that you push a Producer:
…is consumed from your Kafka Consumer:
Conclusion
This advancement in Kafka’s architecture eliminates the dependency on ZooKeeper, introducing a more streamlined and efficient approach to managing Kafka clusters. By leveraging the KRaft mode, users can fully exploit the capabilities of Apache Kafka while enjoying simplified management and increased performance.
Found the article helpful? if so please follow us on Socials
