Apache Kafka – Topics using CLI

In Apache Kafka, a topic is a category or stream of messages that the Kafka message broker (or cluster) stores. Producers write data to topics and consumers read from topics. A Topic in Kafka is similar to a table in a database or a stream in a stream processing system. Each topic is divided into a number of partitions, which allows for parallel processing of the data by consumers. Messages are assigned to partitions based on a partitioning key, which can be any field in the message. Topics are used to store and transmit data in Kafka. They can be used for a wide range of applications, such as real-time data processing, event-driven architectures, and log aggregation. A topic in Kafka can be created, deleted, described, or changed using the CLI. Make sure Kafka is open before you begin. The necessary parameters must be specified in order to describe a Kafka topic and obtain partition information.

• Use the hostname and port for Kafka v2.2+, for example, localhost:9092.
• Use localhost:2181 as the Zookeeper URL and port if you’re using a previous version of Kafka.

Create a Kafka Topic

• The name of the topic, number of partitions, and replication factor are required parameters.
• Use the “–create” option in the CLI kafka-topics.sh.

To create a Kafka topic called “topic1” with 3 partitions and a replication factor of 1, connect to the Kafka broker running at localhost:9092.

kafka-topics.sh --bootstrap-server localhost:9092 --topic topic1 --create --partitions 3 --replication-factor 1

Output:

Creating kafka topic

Additional important Parameters You Can Set (Advanced)

--config

You may configure topics at the topic level, using a command like –config max.message.bytes=64000.

--disable-rack-aware

Disable rack-aware replica assignment (not advised; only set if you are confident in your ability to use it)

List Kafka Topic

Use the “–list” option in the CLI for kafka-topics.sh. Listing topics when my Kafka broker is running at localhost:9092

kafka-topics.sh --bootstrap-server localhost:9092 --list

Output:

Listing kafka topics

Describe a Kafka Topic

In the CLI kafka-topics.sh, use the “–describe“ option. Describing topics when my Kafka broker is running at localhost:9092

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic topic1

Output:

Describing a kafka topic

These are some of the advanced parameters that can be used with “–list” and “–describe.”

--exclude-internal

When using the list or describe commands, ignore internal subjects (they are by default listed).

Using replication status as a filter for topics:

--at-min-isr-partitions

If enabled, only display partitions whose isr count is equal to the given minimum while explaining subjects.

--unavailable-partitions

Only display divisions with an unavailable leader.

--under-min-isr-partitions

A partition will only be shown if its isr count is below the defined minimum.

--under-replicated-partitions

only display the under-replicated partitions.