Spring Boot - Integration with Kafka

Apache Kafka is a distributed event-streaming platform used for building real-time, scalable, and fault-tolerant systems. Spring Boot provides seamless integration with Kafka through Spring for Apache Kafka, making it easy to produce and consume messages in microservices architectures.

Core Concepts of Kafka

Kafka operates as a distributed publish-subscribe messaging system. It enables decoupled communication between data producers and consumers.

• Producer: Publishes messages to Kafka topics.
• Consumer: Subscribes to topics and processes incoming messages.
• Topic: Logical channel where messages are stored. Topics can have multiple partitions for scalability.
• Broker: Kafka server responsible for storing and delivering messages.
• Zookeeper / KRaft: Manages cluster coordination and metadata. (Zookeeper is optional in Kafka ≥ 2.8, replaced by KRaft mode.)

Step-by-Step: Integrating Kafka with Spring Boot

Step 1: Create a Spring Boot Project

Generate a new Spring Boot project using Spring Initializr or IntelliJ IDEA with the following options:

• Name: spring-boot-kafka-example
• Dependencies: Spring Web, Spring for Apache Kafka, DevTools

Step 2: Dependencies

Make sure the following dependencies contain in your pom.xml :

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

Project Structure

Step 3: Configure Kafka Connection

In src/main/resources/application.properties:

spring.application.name=spring-boot-kafka-example
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=group_id
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

These configurations define Kafka broker address, consumer group, and serialization format for messages.

Step 4: Kafka Consumer Configuration

Create the KafkaConsumerConfig class and this configuration class sets up the Kafka consumer of the Spring Boot application.

config/KafkaConsumerConfig.java:

@EnableKafka
@Configuration
public class KafkaConsumerConfig {

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "group_id");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return new DefaultKafkaConsumerFactory<>(props);
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String>
    kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }
}

• @EnableKafka enables Kafka listener detection.
• ConsumerFactory sets up deserializers and connection settings.
• ConcurrentKafkaListenerContainerFactory manages Kafka listener threads.

Step 5: Kafka Producer Configuration

config/KafkaProducerConfig.java:

@Configuration
public class KafkaProducerConfig {

    @Bean
    public ProducerFactory<String, String> producerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        return new DefaultKafkaProducerFactory<>(props);
    }

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }
}

• ProducerFactory defines Kafka producer properties and serializers.
• KafkaTemplate is the key Spring abstraction used to send messages to Kafka topics.

Step 6: Kafka Producer Service

service/KafkaProducerService.java:

@Service
public class KafkaProducerService {

    private static final String TOPIC = "my_topic";
    private final KafkaTemplate<String, String> kafkaTemplate;

    public KafkaProducerService(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void sendMessage(String message) {
        kafkaTemplate.send(TOPIC, message);
        System.out.println("Message sent: " + message);
    }
}

• The KafkaTemplate object sends messages to Kafka.
• Messages are sent to the topic my_topic with a simple log confirmation.

Step 7: Kafka Consumer Service

service/KafkaConsumerService.java:

package com.gfg.springbootkafkaexample.service;

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

@Service
public class KafkaConsumerService {

    @KafkaListener(topics = "my_topic", groupId = "group_id")
    public void consume(String message) {
        System.out.println("Message received: " + message);
    }
}

• @KafkaListener automatically subscribes this method to a Kafka topic.
• Every time a new message arrives, the consume() method is triggered.

Step 8: REST Controller

controller/KafkaController.java

@RestController
public class KafkaController {

    private final KafkaProducerService producerService;

    public KafkaController(KafkaProducerService producerService) {
        this.producerService = producerService;
    }

    @GetMapping("/send")
    public String sendMessage(@RequestParam String message) {
        producerService.sendMessage(message);
        return "Message sent successfully!";
    }
}

• This REST endpoint /send?message=HelloKafka triggers message sending.
• It interacts with the KafkaProducerService to publish data to Kafka.

Step 09: Start the Kafka and Zookeeper

Make sure both services are running:

# Start Zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

# Start Kafka Broker

bin/kafka-server-start.sh config/server.properties

Refer to this article to know how to start both services: How to Install and Run Apache Kafka on Windows?

Step 10: Run the Application

Once completed the project, it will start and run at port 8080.

Step 11: Test the Endpoint

Use Postman or curl to send a request:

GET: http://localhost:8080/send?message=HelloKafka

Response:

Console Logs:

Important Notes

• Kafka and Zookeeper must be running before starting the app
• Producer and consumer must use the same topic
• Topic must exist or auto-creation should be enabled
• application.properties is the preferred place for Kafka configs