Flume系列：Flume常用采集通道案例

iwester

已于 2023-06-06 22:46:47 修改

阅读量1.2k

点赞数

CC 4.0 BY-SA版权

分类专栏： # Flume Hadoop生态文章标签： flume 大数据 hadoop

于 2023-05-26 20:02:01 首次发布

本文链接：https://blog.csdn.net/web_snail/article/details/130893210

本文介绍了Apache Flume在大数据生态中的应用，详细阐述了三种Flume采集通道的配置与使用：从文件到Kafka，以及两种从Kafka到HDFS的方案。内容包括每个通道的启动过程、优缺点分析，如性能和拦截器需求。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Apache Hadoop生态-目录汇总-持续更新

1: 文件->flume->kafka

2: kafka->flume->hdfs (方案一)

3: kafka->flume->hdfs (方案二)

Apache Hadoop生态-目录汇总-持续更新

系统环境：centos7

Java环境：Java8

1: 文件->flume->kafka

tailDir source -> kafka channel

配置文件：file_flume_kafka.conf

# 1：定义组件
file_flume_kafka.sources = r1
file_flume_kafka.channels = c1

# 2：定义source
file_flume_kafka.sources.r1.type = TAILDIR
file_flume_kafka.sources.r1.positionFile = /usr/local/flume-1.9.0/project_v4/tail_dir.json
file_flume_kafka.sources.r1.fileSuffix = .COMPLETED
file_flume_kafka.sources.r1.filegroups = f1
file_flume_kafka.sources.r1.filegroups.f1 = /log/app.*.log
 ## 定义source拦截器(ETL数据清洗，判断数据是否完整)
file_flume_kafka.sources.r1.interceptors = i1
file_flume_kafka.sources.r1.interceptors.i1.type = com.wester.flume.interceptor.ETLInterceptor$Builder

# 3：定义channel
file_flume_kafka.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
file_flume_kafka.channels.c1.kafka.bootstrap.servers = 192.168.5.103:9092,192.168.5.87:9092,192.168.5.114:9092
file_flume_kafka.channels.c1.kafka.topic = project_v4_topic_log
    ## 设置消费者组,保证每次消费时能够获取上次对应的Offset
file_flume_kafka.channels.c1.consumer.group.id = file_flume_kafka
    #设置不用flume组装的header
file_flume_kafka.channels.c1.parseAsFlumeEvent = false

# 4：定义sink - 这里不用

# 5：定义关联关系
file_flume_kafka.sources.r1.channels = c1

启动

nohup /usr/local/flume-1.9.0/bin/flume-ng agent \
  --conf-file /shucang_v4/project_files/flume_jobs/conf/file_flume_kafka.conf \
  -C /shucang_v4/project_files/flume_jobs/jar/project_v4_flume.jar \
  --name file_flume_kafka -Dflume.root.logger=INFO,LOGFILE >/usr/local/flume-1.9.0/logs/file_flume_kafka.log 2>&1  &