sun

一.简介 Flume是Cloudera提供的一个高可用的，高可靠的，分布式的海量日志采集、聚合和传输的系统，Flume支持在日志系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。 image.png 二.主要功能 1.日志收集 Flume最早是Cloudera提供的日志收集系统，目前是Apac

1.Memory Channel a1.channels = c1 a1.channels.c1.type = memory a1.channels.c1.capacity = 10000 a1.channels.c1.transactionCapacity = 10000 a1.channels.c1.byteCapacityBufferPercentage = 20 a1.channel

1.Hdfs Sink a1.channels = c1 a1.sinks = k1 a1.sinks.k1.type = hdfs a1.sinks.k1.channel = c1 a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S a1.sinks.k1.hdfs.filePrefix = events- a1.sinks.k1.

Log4J Appender Appends Log4j events to a flume agent’s avro source. A client using this appender must have the flume-ng-sdk in the classpath (eg, flume-ng-sdk-1.8.0-SNAPSHOT.jar). Required properties

Flume Interceptors Flume has the capability to modify/drop events in-flight. This is done with the help of interceptors. Interceptors are classes that implement org.apache.flume.interceptor.Interce

Flume Channel Selectors If the type is not specified, then defaults to “replicating”. Replicating Channel Selector (default) Required properties are in bold. Property Name Defau

# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyrigh

1.下载 Flume client sdk http://central.maven.org/maven2/org/apache/flume/flume-ng-sdk/1.7.0/flume-ng-sdk-1.7.0.jar 2、RPC client interface Flume的RpcClient实现了Flume的RPC机制。用户的应用程序可以很简单的调用Flume Clie

./flume-ng agent --conf ../conf -f ../conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console

1.  Avro Source属性说明 a1.sources = r1 a1.channels = c1 a1.sources.r1.type = avro a1.sources.r1.channels = c1 a1.sources.r1.bind = 0.0.0.0 a1.sources.r1.port = 4141 2. Spooling Directory Source