Fluentd meetup dive into fluent plugin (outdated)

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Dive into Fluent P
lugin

2012 2 4

Site:
repeatedly.github.com

Company:
Preferred Infrastructure, Inc.

Love plugins:
...

What's Fluentd?
The missing log collector
See keynote

...

Fluentd is a
buffer
router
collector
converter
aggregator
etc...
2012 2 4

... but,
Fluentd doesn’t have such features as a built-in.

2012 2 4

Instead,
Fluentd has flexible plugin architecture
which consists of Input, Output and Buffer.

2012 2 4

We can customize Fluentd using plugins :)

2012 2 4

Agenda
Yes, I talk about
- an example of Fluentd plugins
- Fluentd and librar...

Example
based on bit.ly/fluentd-with-mongo

2012 2 4

Install

Plugin name is fluent-plugin-xxx ,
and fluent-gem is included in Fluentd gem.

2...

Let’s type!

$ fluent-gem install fluent-plugin-mongo

2012 2 4

Me!

Many
Many
Many
Plugins!
2012 2 4

fluentd.conf

Input Output
<source> <match mongo.**>
type tail ...

Start!

$ fluentd -c fluentd.conf
2012-02-04 00:00:14 +0900: starting fluentd-0.10.8
...

Attack!

$ ab -n 100 -c 10 http://localhost/

2012 2 4

$ mongo --host otherhost
> use apache
> db.access.find()
{
"type": "127.0....

Apache I’m a log!

tail

write

...

Warming up

2012 2 4

Fluentd Stack
Output
Input
Buffer
...

Ruby
ruby-lang.org

2012 2 4

Fluentd and plugins are written in Ruby.

2012 2 4

... but note that
Fluentd works on Ruby 1.9, goodbye 1.8!

2012 2 4

MessagePack
msgpack.org

2012 2 4

Serialization:
JSON like fast and compact format.

RPC:
Async and parallelism for high performance....

Binary format,
Header + Body,
and
Variable length.

2012 2 4

Note that
Ruby version can’t handle a Time object.

2012 2 4

So,
we use an Integer object instead of a Time.

2012 2 4

Source:
github.com/msgpack

Wiki:
wiki.msgpack.org/display/MSGPACK

Mailing List:
g...

Cool.io
coolio.github.com

2012 2 4

Event driven framework built on top of libev.

2012 2 4

Cool.io has Loop and Watchers with
Transport wrappers.

2012 2 4

Fluentd has a default event loop.
We can use @default_loop in the plugin.

2012 2 4

Configuration

2012 2 4

Fluentd loads plugins from $LOAD_PATH.

2012 2 4

Input:
$LOAD_PATH/fluent/plugin/in_<type>.rb

Buffer:
$LOAD_PATH/fluent/plugin/buf_<type>.rb

...

We use ‘register_input’, ‘register_buffer’
and ‘register_output’ to register a plugin.

2012 2 4

We can load the plugin configuration using
config_param and configure method.
config_param s...

<source>
type tail
path /path/to/log
...
</source> ...

One trick is here:

Fluentd’s configuration module does not
verify a default value. So,
...

Fluentd provides some useful mixins for
input and output plugins.

2012 2 4

SetTagKeyMixin:
Provide ‘tag_key’ and ‘include_tag_key’.

SetTimeKeyMixin:
Provide ‘time_key’ and ‘...

Mixin usage
Code Flow
super
class Mong...

Input

2012 2 4

Available plugins

Default 3rd party
exec mongo_tail
forward ...

class NewInput < Input
...
def configure(conf)
# parse a configuration m...

In action method,
we use Engine.emit to input data.

tag = "app.tag"
...

How to read an input in an efficient way?
We use a thread and an event loop.

2012 2 4

Thread
class ForwardInput < Fluent::Input
...
def start
....

Event loop
class ForwardInput < Fluent::Input
...
def start
...

Note that
We must use Engine.now instead of Time.now

2012 2 4

Buffer

2012 2 4

Available plugins

Default 3rd party

memory
file
zfile (?...

In most cases,
Memory and File are enough.

2012 2 4

Memory type is default.
It’s fast but can’t resume data.

2012 2 4

File type is persistent type.
It can resume data from file.

2012 2 4

Output

2012 2 4

Available plugins

Default 3rd party
copy mongo
exec s3...

class NewOutput < BufferedOutput
# configure, start and shutdown
# are same as input plugi...

Output has 3 buffering modes.
None
Buffered
...

Buffering type

Buffered Time sliced
from in
Buffer has an int...

How to write an output in an efficient way?
We can use multi-process (input too).

See: Deta...

Test

2012 2 4

Input:
Fluent::Test::InputTestDriver

Buffer:
Fluent::Test::BufferedOutputTestDriver

Outp...

class MongoOutputTest < Test::Unit::TestCase
def setup
Fluent::Test.setup
...

...
def test_format
# test format using emit and expect_format
end
...

It’s a weak point in Fluentd... right?

2012 2 4

Release

2012 2 4

Gem Structure
Plugin root
|-- lib/
| |-- fluent/
| |-- ...

Bundle with git
$ edit lib/fluent/plugin/out_<name>.rb

$ git add / commit

$...

See released plugins
for more details about each file.

2012 2 4

Lastly...

2012 2 4

Help!
2012 2 4

Question?

2012 2 4

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Fluentd Hacking Guide at RubyKaigi ... by Naotoshi Seo 77299 views
Life of an Fluentd event by Kiyoto Tamura 587691 views
Dive into Fluentd plugin v0.12 by N Masahiro 25252 views
Fluentd meetup #2 by Treasure Data, Inc. 9723 views
Big Data入門に見せかけたFluentd入門 by Keisuke Takahashi 31330 views
Fluentd v0.14 Overview by N Masahiro 6888 views

View on Slideshare

1 of 73 Ad

View on Slideshare

1 of 73 Ad