Kafka Automatic Partitioning Guide - Using test-topic as an Example

Let Kafka auto-generate the reassignment structure

The current partition status of test-topic shows that it has 5 partitions, each with only 1 replica (ReplicationFactor=1), which does not align with best practices for a production environment. Below, I will provide a detailed guide on how to automatically expand the number of partitions and increase the replication factor for test-topic.

root@elk-105:~# /opt/kafka/bin/kafka-topics.sh --describe --bootstrap-server 192.168.126.104:9092 --topic test-topic
[2025-04-06 07:54:21,935] WARN [AdminClient clientId=adminclient-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics. (org.apache.kafka.clients.admin.KafkaAdminClient)
Topic: test-topic       TopicId: 6MNTgH95TEiAhMyxAj0tPA PartitionCount: 5       ReplicationFactor: 1    Configs: segment.bytes=1073741824
        Topic: test-topic       Partition: 0    Leader: 1       Replicas: 1     Isr: 1  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 1    Leader: 2       Replicas: 2     Isr: 2  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 2    Leader: 3       Replicas: 3     Isr: 3  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 3    Leader: 1       Replicas: 1     Isr: 1  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 4    Leader: 2       Replicas: 2     Isr: 2  Elr: N/A        LastKnownElr: N/A


root@elk-105:~# /opt/kafka/bin/kafka-topics.sh --alter --bootstrap-server 192.168.126.104:9092 --partitions 10 --topic test-topic
[2025-04-06 07:54:29,669] WARN [AdminClient clientId=adminclient-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics. (org.apache.kafka.clients.admin.KafkaAdminClient)


root@elk-105:~# /opt/kafka/bin/kafka-topics.sh --describe --bootstrap-server 192.168.126.104:9092 --topic test-topic
[2025-04-06 07:56:40,040] WARN [AdminClient clientId=adminclient-1] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics. (org.apache.kafka.clients.admin.KafkaAdminClient)
Topic: test-topic       TopicId: 6MNTgH95TEiAhMyxAj0tPA PartitionCount: 10      ReplicationFactor: 1    Configs: segment.bytes=1073741824
        Topic: test-topic       Partition: 0    Leader: 1       Replicas: 1     Isr: 1  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 1    Leader: 2       Replicas: 2     Isr: 2  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 2    Leader: 3       Replicas: 3     Isr: 3  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 3    Leader: 1       Replicas: 1     Isr: 1  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 4    Leader: 2       Replicas: 2     Isr: 2  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 5    Leader: 3       Replicas: 3     Isr: 3  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 6    Leader: 1       Replicas: 1     Isr: 1  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 7    Leader: 2       Replicas: 2     Isr: 2  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 8    Leader: 3       Replicas: 3     Isr: 3  Elr: N/A        LastKnownElr: N/A
        Topic: test-topic       Partition: 9    Leader: 1       Replicas: 1     Isr: 1  Elr: N/A        LastKnownElr: N/A

Step one: of course. Below is the topics-to-move.json file content for your signal-topic-2025:

root@elk-105:~/kafka-replication# cat topics-to-move.json
{
  "version": 1,
  "topics": [
    {
      "topic": "test-topic"
    }
  ]
}

Step two:  generate a new replica reassignment plan:

root@elk-105:~/kafka-replication# /opt/kafka/bin/kafka-reassign-partitions.sh \
>   --bootstrap-server 192.168.126.104:9092 \
>   --broker-list 1,2,3 \
>   --topics-to-move-json-file topics-to-move.json \
>   --generate

The generated plan will be displayed in the terminal. You can save it as a JSON file and execute it:

root@elk-105:~/kafka-replication# /opt/kafka/bin/kafka-reassign-partitions.sh \
>   --bootstrap-server 192.168.126.104:9092 \
>   --broker-list 1,2,3 \
>   --topics-to-move-json-file topics-to-move.json \
>   --generate
[2025-04-06 08:00:46,867] WARN [AdminClient clientId=reassign-partitions-tool] The DescribeTopicPartitions API is not supported, using Metadata API to describe topics. (org.apache.kafka.clients.admin.KafkaAdminClient)
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test-topic","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"test-topic","partition":1,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":2,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":3,"replicas":[1],"log_dirs":["any"]},{"topic":"test-topic","partition":4,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":5,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":6,"replicas":[1],"log_dirs":["any"]},{"topic":"test-topic","partition":7,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":8,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":9,"replicas":[1],"log_dirs":["any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test-topic","partition":0,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"test-topic","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":3,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":4,"replicas":[1],"log_dirs":["any"]},{"topic":"test-topic","partition":5,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":6,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":7,"replicas":[1],"log_dirs":["any"]},{"topic":"test-topic","partition":8,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":9,"replicas":[3],"log_dirs":["any"]}]}

Output:

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test-topic","partition":0,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"test-topic","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":3,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":4,"replicas":[1],"log_dirs":["any"]},{"topic":"test-topic","partition":5,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":6,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":7,"replicas":[1],"log_dirs":["any"]},{"topic":"test-topic","partition":8,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":9,"replicas":[3],"log_dirs":["any"]}]}

Step three: certainly, below is the Proposed partition reassignment configuration you just generated, formatted clearly and properly as JSON. You can save it directly as signal-topic-new-reassignment.json for use:

{
  "version": 1,
  "partitions": [
    {
      "topic": "test-topic",
      "partition": 0,
      "replicas": [3],
      "log_dirs": ["any"]
    },
    {
      "topic": "test-topic",
      "partition": 1,
      "replicas": [1],
      "log_dirs": ["any"]
    },
    {
      "topic": "test-topic",
      "partition": 2,
      "replicas": [2],
      "log_dirs": ["any"]
    },
    {
      "topic": "test-topic",
      "partition": 3,
      "replicas": [3],
      "log_dirs": ["any"]
    },
    {
      "topic": "test-topic",
      "partition": 4,
      "replicas": [1],
      "log_dirs": ["any"]
    },
    {
      "topic": "test-topic",
      "partition": 5,
      "replicas": [2],
      "log_dirs": ["any"]
    },
    {
      "topic": "test-topic",
      "partition": 6,
      "replicas": [3],
      "log_dirs": ["any"]
    },
    {
      "topic": "test-topic",
      "partition": 7,
      "replicas": [1],
      "log_dirs": ["any"]
    },
    {
      "topic": "test-topic",
      "partition": 8,
      "replicas": [2],
      "log_dirs": ["any"]
    },
    {
      "topic": "test-topic",
      "partition": 9,
      "replicas": [3],
      "log_dirs": ["any"]
    }
  ]
}

Step four: save the above as signal-topic-new-reassignment.json, then execute:

/opt/kafka/bin/kafka-reassign-partitions.sh \
  --bootstrap-server 192.168.126.104:9092 \
  --reassignment-json-file signal-topic-new-reassignment.json \
  --execute

Step five: once completed, you can verify if it's done using:

/opt/kafka/bin/kafka-reassign-partitions.sh \
  --bootstrap-server 192.168.126.104:9092 \
  --reassignment-json-file signal-topic-new-reassignment.json \
  --verify