kafka从指定位置重播

老黄瓜001

于 2020-12-11 12:00:52 发布

阅读量738

点赞数

CC 4.0 BY-SA版权

分类专栏： kafka 文章标签： java kafka

本文链接：https://blog.csdn.net/qq_33894426/article/details/111032682

kafka 专栏收录该内容

1 篇文章

订阅专栏

本文详细探讨了Kafka如何实现消息重播，以及消费者如何从特定偏移量开始读取数据。通过深入理解Kafka的分区和偏移管理，读者将能够配置消费者组以重新处理已消费的消息，这对于测试、数据处理和故障恢复场景非常有用。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

.....
//配置
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
List<PartitionInfo> partitions = consumer.partitionsFor("test-topic");
List<TopicPartition> topicPartitions = Lists.newArrayList();
for (PartitionInfo partitionInfo : partitions) {
    TopicPartition topicPartition = new TopicPartition("test-topic",partitionInfo.partition());
    topicPartitions.add(topicPartition);
}
consumer.assign(topicPartitions);
Set<TopicPartition> assignments = consumer.assignment();
/* 构造offsetsForTimes参数，通过时间戳找到offset */
Map<TopicPartition, Long> query = Maps.newHashMap();
for (TopicPartition topicPartition : assignments) {
    query.put(topicPartition, startTime);
}
Map<TopicPartition, OffsetAndTimestamp> result = consumer.offsetsForTimes(query);
for (Map.Entry<TopicPartition, OffsetAndTimestamp> entry : result.entrySet()) {
    // 每个topic的partition都seek到执行的offset
    consumer.seek(entry.getKey(), entry.getValue().offset());
}
while(true){    
    ConsumerRecords<String, String> consumerRecord =consumer.poll(Duration.ofMillis(100));
    //then do your bussines

}