How to monitor Kafka using JMX

by

This article discusses how you can actively monitor Apache Kafka using JMX and the JmxTool that is available in your installation. At the end of this tutorial, you will be able to collect key metrics to troubleshoot and monitor your Kafka cluster.

Kafka is a distributed streaming platform that is widely used for building real-time data pipelines and streaming applications. As with any distributed system, monitoring Kafka is crucial for ensuring its availability, reliability, and performance. Fortunately, Kafka provides a rich set of metrics that can be monitored via Java Management Extensions (JMX).

JMX is a Java-based technology that provides a standard way of monitoring and managing Java applications. Kafka exposes its metrics via JMX, which means that you can use any JMX-compliant monitoring tool to collect and visualize Kafka metrics. One such tool is the kafka.tools.JmxTool.

Prerequisites

Before you can use the kafka.tools.JmxTool to monitor Kafka, you need to make sure that JMX is enabled in your Kafka brokers. You can enable JMX by setting the following properties in your Kafka broker configuration file (server.properties):

# Enable JMX
com.sun.management.jmxremote=true

# Set JMX port (default is 9999)
com.sun.management.jmxremote.port=<jmx_port>

# Set JMX authentication (optional)
com.sun.management.jmxremote.authenticate=true
com.sun.management.jmxremote.password.file=<password_file>
com.sun.management.jmxremote.access.file=<access_file>

You can also enable JMX by setting the following environment property, before starting the Broker:

export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

Using the kafka.tools.JmxTool

The kafka.tools.JmxTool is a command-line tool that allows you to query Kafka metrics via JMX. To use the tool, follow these steps:

Download the Kafka binaries from the Kafka website and extract them to a directory on your monitoring machine.

Navigate to the bin/ directory of the extracted Kafka binaries.

Run the following command to launch the kafka.tools.JmxTool:

./kafka-run-class.sh kafka.tools.JmxTool <jmx_url> <metric_name>

Replace <jmx_url> with the JMX URL of your Kafka broker, which is in the format service:jmx:rmi:///jndi/rmi://<hostname>:<jmx_port>/jmxrmi. Replace <metric_name> with the name of the Kafka metric you want to monitor, along with its corresponding JMX object name (e.g., kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec).

The kafka.tools.JmxTool will output the current value of the specified metric.

Sample JMX Metrics you can collect

Here is a list of some common JMX Metrics you can collect from your Kafka Broker:

  1. kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec: The number of bytes that Kafka brokers receive from producers and consumers each second.
  2. kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec: The number of bytes being sent by Kafka brokers to consumers and other brokers each second.
  3. kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec: The number of bytes being rejected by Kafka brokers due to insufficient broker resources.
  4. kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=*: The lag of a consumer group with the given clientId for each partition in each topic. (Note that * can be replaced with the actual clientId).
  5. kafka.server:type=Produce,topic= ,name=FailedProduceRequestsPerSec:* The number of produce requests that failed due to errors on the broker or the producer for a specific topic.
  6. kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec: The number of messages that Kafka brokers receive from producers and consumers each second.
  7. kafka.server:type=BrokerTopicMetrics,name=MessagesOutPerSec: The number of messages being sent by Kafka brokers to consumers and other brokers each second.
  8. kafka.server:type=Partition,name=PartitionCount,topic=*: The total number of partitions in a specific topic.
  9. kafka.controller:type=KafkaController,name=UnderReplicatedPartitions: The number of partitions whose replicas are not up-to-date with the leader replica.
  10. kafka.network:type=RequestChannel,name=RequestQueueSize,request=Fetch: The size of the request queue for fetch requests waiting to be processed.

As an example, here is how to collect metrics for the BytesInPerSec:

bin/kafka-run-class.sh kafka.tools.JmxTool --one-time true --object-name kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec --report-format csv
Trying to connect to JMX url: service:jmx:rmi:///jndi/rmi://:9999/jmxrmi.
time,"1677140018882"
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec:Count,"0"
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec:EventType,"bytes"
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec:FifteenMinuteRate,"0.0"
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec:FiveMinuteRate,"0.0"
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec:MeanRate,"0.0"
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec:OneMinuteRate,"0.0"
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec:RateUnit,"SECONDS"

Conclusion

In this tutorial, you learned how to monitor Kafka with JMX using the kafka.tools.JmxTool. With this tool, you can easily query Kafka metrics via JMX and gain insights into the health, performance, and behavior of your Kafka brokers. By monitoring Kafka metrics on a regular basis, you can detect and diagnose issues early on, optimize performance, and ensure the reliability of your Kafka-based applications.