1. Introduction
In this tutorial, we’ll take a look at different ways of retrieving the total number of partitions of a Kafka Topic. After a short introduction about what is a Kafka Partition and why we might need to retrieve this information, we’ll write the Java code to execute this operation. Then we’ll see how it’s possible to get this information using CLI.
2. Kafka Partition
A Kafka Topic can be divided into several partitions. The goal of having more than one partition is to be able to consume messages from the same topic concurrently. Because it’s useless to have more consumers than existing partitions, the number of Kafka Partitions in a topic represents the maximum parallelism level for consuming. So knowing in advance how many partitions a given topic has can be useful for sizing the respective consumers correctly.
3. Retrieve the Partition Number using Java
To retrieve the number of partitions for a specific topic using Java, we can rely on the KafkaProducer.partitionFor(topic) method. This method will return the partition metadata for the given topic:
Properties producerProperties = new Properties();
// producerProperties.put("key","value") ...
KafkaProducer<String, String> producer = new KafkaProducer<>(producerProperties)
List<PartitionInfo> info = producer.partitionsFor(TOPIC);
Assertions.assertEquals(3, info.size());
The size of the PartitionInfo list returned by this method will be exactly equal to the number of partitions configured for the specific topic.
If we don’t have access to a Producer, we can achieve the same results using Kafka AdminClient in a slightly more complex way:
Properties props = new Properties();
// props.put("key","value") ...
AdminClient client = AdminClient.create(props)){
DescribeTopicsResult describeTopicsResult = client.describeTopics(Collections.singletonList(TOPIC));
Map<String, KafkaFuture> values = describeTopicsResult.values();
KafkaFuture topicDescription = values.get(TOPIC);
Assertions.assertEquals(3, topicDescription.get().partitions().size());
In this case, we are relying on the KafkaClient.describeTopic(topic) method, which returns a DescribeTopicsResult object containing a Map of future tasks to be executed. Here we are retrieving only the TopicDescription of the Topic we need and, finally, retrieving the number of partitions.
4. Retrieve the Partition Number using CLI
We have a couple of options to retrieve the number of partitions of a given Topic with CLI.
First of all, we can rely on the shell scripts that come with every Kafka install and run:
$ kafka-topics --describe --bootstrap-server localhost:9092 --topic topic_name
This command will output a complete description of the topic specified:
Topic:topic_name PartitionCount:3 ReplicationFactor:1 Configs: ...
Another option is to use Kafkacat, a non-JVM-based consumer and producer for Kafka. With the metadata list mode (-L), this shell utility shows the current state of the Kafka cluster, including all its topics and partitions. To show metadata information of a specific topic, we can run the following:
$ kafkacat -L -b localhost:9092 -t topic_name
The output of this command will be:
Metadata for topic topic_name (from broker 1: mybroker:9092/1):
topic "topic_name" with 3 partitions:
partition 0, leader 3, replicas: 1,2,3, isrs: 1,2,3
partition 1, leader 1, replicas: 1,2,3, isrs: 1,2,3
partition 2, leader 1, replicas: 1,2, isrs: 1,2
We can see how this shell utility command also shows useful details about a specific topic and its partitions.
5. Conclusion
In this short tutorial, we have seen how it’s possible to retrieve the total number of partitions of a specific Kafka Topic using Java and CLI.
We have first seen why retrieving this information is useful, and then we used KafkaProducer and KafkaAdmin. Finally, we have used shell commands with Kafka script utilities and KafkaCat.
As always, the full source code of the article is available over on GitHub.
