Spring Boot - Create and Configure Topics in Apache Kafka

In Apache Kafka, topics are used to organize and store messages. They act as logical channels where producers send messages and consumers read them. In a Spring Boot application, Kafka topics can be created and configured programmatically so that they are automatically generated when the application starts.

• Topic Organization: Kafka topics store and organize messages in a sequential log structure.
• Automatic Creation: Spring Boot can automatically create topics when the application starts.
• Configuration Flexibility: Developers can configure topic properties like partitions and replicas.

Step-by-Step Implementation

Step 1: Create Spring Boot Project

Go to Spring Initializr and create a new Spring Boot project.

• Project: Maven
• Language: Java
• Choose the latest Spring Boot version.
• Add the following dependency: Spring for Apache Kafka
• Click Generate and download the project.
• Extract the project and open it in your IDE (IntelliJ / Eclipse / STS).

If the dependency was not added during project creation, add it manually in the pom.xml file.

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>2.9.0</version>
    <scope>compile</scope>
</dependency>

Step 2: Create Topic Configuration Class

Create a new class named TopicConfiguration.java.Annotate the class with @Configuration.

• This class will contain Kafka topic configuration.
• @Configuration tells Spring Boot that this class contains bean definitions.
• Spring Boot automatically detects and loads configuration classes

import java.io.*;

@Configuration
public class TopicConfiguration {
  
}

Step 3: Configure KafkaAdmin Bean

Create a KafkaAdmin bean to connect the application with the Kafka broker. The next step is to construct a new bean that tells KafkaAdmin what our Kafka URL is.

BOOTSTRAP_SERVERS_CONFIG: Host and port on where Kafka broker is running.

BOOTSTRAP_SERVERS_CONFIG: <kafka_url>:<port>

@Bean 
public KafkaAdmin admin()
{
    Map<String, Object> configs = new HashMap<>();
    configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
                "localhost:9092");
    return new KafkaAdmin(configs);
}

Step 4: Create Kafka Topic

Now create a topic using TopicBuilder. A lot of various settings can be used right here while creating a topic.

import java.io.*;

@Bean 
public NewTopic topic1()
{
    return TopicBuilder.name("TOPIC-1")
        .partitions(1)
        .replicas(1)
        .build();
}

@Bean
public NewTopic topic2()
{
    return TopicBuilder.name("TOPIC-2")
        .partitions(1)
        .replicas(1)
        .build();
}

File: TopicConfiguration.java

import java.io.*;

@Configuration
public class TopicConfiguration {

    @Bean 
      public KafkaAdmin admin()
    {
        Map<String, Object> configs = new HashMap<>();
        configs.put(
            AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
            "localhost:9092");
        return new KafkaAdmin(configs);
    }

    @Bean 
      public NewTopic topic1()
    {
        return TopicBuilder.name("TOPIC-1")
            .partitions(1)
            .replicas(1)
            .build();
    }

    @Bean 
      public NewTopic topic2()
    {
        return TopicBuilder.name("TOPIC-2")
            .partitions(1)
            .replicas(1)
            .build();
    }
}

Step 6: Run the Spring Boot Application

Run the main Spring Boot application class.

• Make sure Kafka server and Zookeeper are running before starting the application.
• When the application starts, the topic will be created automatically.
• If the topic already exists, Kafka will ignore the creation request.

Use this command:

kafka-topics.sh --bootstrap-server localhost:9092 --topic <topic_name> --describe

Output:

This is the output from the server.