Downloaded 71 times
![1) Support System
• All logs are stored in MongoDB
• No need to write ad-hoc log analysis scripts.
before after
#!/usr/bin/ruby $ mongo i-hate-mongo
open( log.txt ).each_line {¦l¦ > db.logs.find({ user : 12345})
ls = l.strip.split
user = ls[4]
puts l if user == 12345
}
$ ruby a.rb > foo.txt](https://image.slidesharecdn.com/fluentdcasual20120518-120518063206-phpapp02/75/Fluentd-for-Small-Projects-Fluentd-Casual-20120518-15-2048.jpg)
Uploaded byTreasure Data, Inc.
Fluentd for Small Projects @ Fluentd Casual 20120518
![Governance, Deployment & Methodologies for Agentic Automation [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day2-251112135038-a386476d-thumbnail.jpg?width=640&height=640&fit=bounds)
Fluentd for Small Projects @ Fluentd Casual 20120518
- 1. Fluentd For Small Projects Structured logging Pluggable architecture Reliable forwarding Kazuki Ohta Treasure Data, Inc. @kzk_mover
- 2. Self-Introduction • Kazuki Ohta > twitter: @kzk_mover > github: kzk • Treasure Data, Inc. > Chief Technology Officer; Founder > Original Fluentd Author @frsyuki is another co-founder. • Open-Source Enthusiast > KDE, uim, Hadoop, memcached, Mozilla, Mongo, etc. > td-agent package manager
- 3.
- 4. What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } Storage record
- 5. What’s Fluentd? Application filter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
- 6. What’s Fluentd? syslogd Scribe Application File Plug-in tail Plug-in Plug-in Fluentd filter / buffer / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
- 7. Fluentd is Casual...? @tagomoris @hotchpotch 15,000,000 UU @ Cookpad ∼80,000 msgs/sec
- 8.
- 9. Fluentd Solves DailyProblems • Operation Problems > ssh root@bad-node && tail -f log.txt > cap deploy && pray > How many PV/UUs we have now? > How many users which did XXX and YYY? • Support > User A claimed that he bought 10 items, could you confirm that?
- 10. But Your Boss Might Say.... • 「Log... For What?」 • 「What’s Fluentd?」 • 「OK, Who uses It?」 • 「We have syslogd」
- 11. Start Fluentd, Silently... • Log Everything As JSON! > TSV/CSV logs are the nightmare. • Install td-agent > rpm/deb/chef/puppet etc. • tail plugin + MongoDB > tailing existing files. > no need to change existing systems.
- 12. Figure 1: CommonLog Operations Analytics Error Notify Recommendation
- 13. Figure 2: Typesof Logs App Log Access Log (Apache, Rails, etc.) System Log (syslog etc.) Others
- 14. 5 ways toconvince your boss to use Fluentd
- 15. 1) Support System •All logs are stored in MongoDB • No need to write ad-hoc log analysis scripts. before after #!/usr/bin/ruby $ mongo i-hate-mongo open( log.txt ).each_line {¦l¦ > db.logs.find({ user : 12345}) ls = l.strip.split user = ls[4] puts l if user == 12345 } $ ruby a.rb > foo.txt
- 16. 2) Realtime SystemStats SCREENSHOT
- 17. 2) Realtime SystemStats • Librato Metrics > Realtime Stats > Realtime Alert > Graphite+Nagios as a Service > GrowthForecast+Nagios http://d.hatena.ne.jp/tagomoris/20120330/1333077746 • Important Metrics.... > queue depth, # of concurrent users, # of sql queries, # of slow queries, etc.
- 18. 3) Error LogCollection # waf.rb App Fluentd MongoDB begin WAF app.run ERROR! rescue => e Fluent::Logger.post("error", {"e"=>e.to_s}) end $ mongo db > db.error.find() # Recent 10 errors
- 19. 4) Error Notifications • “Semi-Realtime” Error Notifications • IRC, HipChat, Campfire, PagerDuty, AirBrake, Skype, etc. TD s Internal Chat <match app.error> type campfire room DevOpsRoom </match>
- 20. 5) Log Analysis MongoDB • MongoDB App Fluentd $ mongo db TD > db.logs.distinct( uid ).count() • Treasure Data $ td query SELECT count(distinct uid) FROM logs
- 21. Conclusion. Fluentd is for... • Support System • Realtime System Stats • Realtime Alert • Error Notifications • Log Analysis • ...
- 22.
- 23.
- 24.
- 25.