The Art Of Application Logging PHPNW12

The Art Of Application Logging PHPNW12

• 1.
  The Art OfApplication Logging PHPNW 2012 Ben Waine - @bwaine http://github.com/benwaine/ApplicationLoggingTalk
• 2.
  Kicking Ass And Logging Names PHPNW 2012 Ben Waine - @bwaine http://github.com/benwaine/ApplicationLoggingTalk
• 3.
  Ben Waine Contractor /Co Founder Recensus PHP Developer Logs in spare time....
• 6.
  Roadmap • Introduction -Why Is Logging Good? • The Mechanics - Code / Examples / Live Demos (GULP) • Logging Strategy - What to do with your logs • Conclusion
• 7.
  Roadmap • Introduction -Why Is Logging Good? • The Mechanics - Code / Examples / Live Demos (GULP) • Logging Strategy - What to do with your logs • Conclusion
• 8.
  Name Some Logs.....
• 9.
  Name Some Logs.....
• 10.
  The Benefits Logs Provide
• 11.
  tail -f
• 12.
  • Insight intohow a process is being executed • Ability to analyse execution after the fact • Ability to catch information about execution errors • Look for patterns and trends
• 13.
  These are all services......
• 14.
  What about application logging?
• 15.
  I SKYBET’s Log
• 20.
  Benefits For Beginners •Provide direction when picking up an application for the first time. • Well sign posted layers increases understanding of the request. • Quick error isolation and context • Times in logs keep performance prominent
• 21.
  Benefits For Everyone
• 23.
  Roadmap • Introduction -Why Is Logging Good? • The Mechanics - Code / Examples / Live Demos (GULP) • Logging Strategy - What to do with your logs • Conclusion
• 24.
  Our Logging Goals Loginformation about a script which creates users. User - Represents a user UserMapper - A class which saves users Scripts - Control User and UserMapper
• 25.
  1) The Basics https://github.com/benwaine/ApplicationLoggingTalk/tree/master/ examples/01-NativeLogging
• 26.
  error_log()
• 27.
  require_once __DIR__ .'/app/User.php'; require_once __DIR__ . '/app/UserMapper.php'; require_once __DIR__ . '/../../vendor/autoload.php'; // Create a mapper $mapper = new UserMapper(); // Create Users and persist them error_log('Beginning User Creation'); while(true) { $user = new User(rand(1, 10000), 'Betty Boo'); $mapper->save($user); sleep(1); unset($user); } 01-basic.php
• 28.
  class UserMapper { public function save(User $user) { error_log('Saving User: ' . $user->getId()); // Code For Saving User } } UserMapper.php
• 29.
  DEMO
• 30.
  require_once __DIR__ .'/app/User.php'; require_once __DIR__ . '/app/UserMapper.php'; require_once __DIR__ . '/../../vendor/autoload.php'; ini_set('error_log', '/tmp/user.log'); // Create a mapper $mapper = new UserMapper(); // Create Users and persist them error_log('Beginning User Creation'); while(true) { $user = new User(rand(1, 10000), 'Betty Boo'); $mapper->save($user); sleep(1); unset($user); } 02-ToFile.php
• 31.
  Pros • Native Functionality Cons •error_log, semantically correct? • PHP errors mixed in with application logging • Not very powerful compared to other tools
• 32.
  2) Logging Library https://github.com/benwaine/ApplicationLoggingTalk/tree/master/ examples/02-LoggingAsADependency
• 33.
  PEARLog Log4PHP Monolog ZendLog KLogger
• 34.
  Monolog https://github.com/Seldaek/monolog
• 35.
  Monolog Architecture Logger use MonologLogger; use MonologHandlerStreamHandler; use MonologHandlerFirePHPHandler; // Create the logger Logger delegates to one $logger = new Logger('my_logger'); or more handlers // Now add some handlers $logger->pushHandler(new StreamHandler(__DIR__.'/my_app.log', Handler Logger::DEBUG)); $logger->pushHandler(new FirePHPHandler()); FirePHPHandler // You can now use your logger $logger->addInfo('My logger is now ready'); Handlers are placed in a stack Log formatting is changed Handler Formatter using supplied or custom StreamHandler (optional) formatters. Handlers can be used by multiple loggers. Formatters can be used by multiple handlers.
• 36.
  Fingers Crossed Logger
• 37.
  // ... RequireStatements ... use MonologLogger; use MonologHandlerStreamHandler; // Create a New MonoLog $logger = new MonologLogger('Application Log'); // Add a handler (writer) $logger->pushHandler(new StreamHandler('/tmp/user.log', Logger::DEBUG)); // Create a mapper $mapper = new UserMapper($logger); // Create Users and persist them $logger->notice('Beginning User Creation'); while(true) { $user = new User(rand(1, 10000), 'Betty Boo'); $mapper->save($user); sleep(1); unset($user); } 01-basic.php
• 38.
  use MonologLogger; class UserMapper{ protected $logger; public function __construct(Logger $logger) { $this->logger = $logger; } public function save(User $user) { $this->logger->info('Saving User: ' . $user->getID()); // Code For Saving User } } UserMapper.php
• 39.
  DEMO
• 40.
  Pros • Object OrientedLogging • Multiple Log Writers (File, Email, Firebug, StdOut) • Log Levels (See Example) Cons • ‘Unwieldy’ instantiation • Logging control code permeates application layer
• 41.
  Log Levels
• 42.
  RFC: 5434 (The SyslogProtocol) 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages
• 43.
  use MonologLogger; use MonologHandlerStreamHandler; $environment= getenv('ENVIRONMENT'); $level = ($environment == "live") ? Logger::NOTICE : Logger::DEBUG; // Create a New MonoLog $logger = new MonologLogger('Application Log'); // Add a handler (writer) $logger->pushHandler(new StreamHandler('/tmp/user.log', $level)); // Create a mapper $mapper = new UserMapper($logger); // Create Users and persist them $logger->notice('Beginning User Creation'); while(true) { $user = new User(rand(1, 10000), 'Betty Boo'); $mapper->save($user); sleep(1); unset($user); 02-LogLevels.php }
• 44.
  use MonologLogger; class UserMapper{ protected $logger; public function __construct(Logger $logger) { $this->logger = $logger; } public function save(User $user) { $this->logger->info('Saving User: ' . $user->getID()); // Code For Saving User } } UserMapper.php
• 45.
  DEMO
• 46.
  3) Your ApplicationLogger https://github.com/benwaine/ApplicationLoggingTalk/tree/master/ examples/03-YourApplicationLogClass
• 47.
  public function __construct(MLogger$application, MLogger $security, array $requestParams) { $this->application = $application; $this->security = $security; $this->requestParams = $requestParams; $this->requestId = md5(microtime() . $this->requestParams['REMOTE_ADDR']); } Logger.php - __constructor
• 48.
  /** * Addsa record to the application log. * * @param string $message * @param int $level * * @return void */ public function applicationLog($message, $level) { // Including a request identifier allows us to track a // users progress throughout the application by grepping the ID. $context = array('RID' => $this->requestId); $this->application->addRecord($level, $message, $context); } Logger.php - applicationLog
• 49.
  // Create firstlogger (application) $applicationLogger = new MonoLogger('Application'); // Add a handler (writer) $applicationLogger->pushHandler(new StreamHandler('/tmp/ application.log', MonoLogger::DEBUG)); // Create second logger (security) $securityLogger = new MonoLogger('Security'); $securityLogger->pushHandler(new StreamHandler('/tmp/security.log', MonoLogger::DEBUG)); $logger = new Logger($applicationLogger, $securityLogger, $requestParams); // Create a mapper $mapper = new UserMapper($logger); 01-basic.php 1/2
• 50.
  // Create Usersand persist them $logger->applicationLog('Beginning User Creation', Logger::INFO); for($x = 0; $x <= 4; $x++) { $user = new User(rand(1, 10000), 'Betty Boo'); // The mapper generates some informational logs when the // the user record is 'saved' $mapper->save($user); unset($user); if($x % 2) { // Simulate some kind of security warning when creating some // of the users. $logger->securityLog('UNSAFE USER!', Logger::WARNING); } } $logger->applicationLog('Ending User Creation', Logger::INFO); 01-basic.php 2/2
• 51.
  DEMO
• 52.
  BEN TURN ONTHE OTHER EXAMPLE NOW. Thanks.
• 53.
  Pros • Consolidated loggingcontrol logic • Encapsulate common logging actions Cons • Still ‘Unwieldy’ instantiation • Can be a little inflexible
• 54.
  4) Your ApplicationLogger (DI’ed) https://github.com/benwaine/ApplicationLoggingTalk/tree/master/ examples/04-DiConfiguredLogging
• 55.
  parameters: logger.class: Logger logger.security.path: /tmp/security.log logger.security.name: security logger.security.level: 200 logger.application.path: /tmp/application.log logger.application.name: application logger.application.level: 200 logger.monolog.class: MonologLogger logger.monolog.handlerClass: MonologHandlerStreamHandler user.mapper.class: UserMapper request.params: REMOTE_ADDR: - 127.0.0.1 services.yml 1/2
• 56.
  services: logger: class: %logger.class% arguments: [@applog, @seclog, %request.params%] seclog: class: %logger.monolog.class% arguments: [%logger.security.name%] calls: - [ pushHandler, [ @seclogHandler ] ] applog: class: %logger.monolog.class% arguments: [%logger.application.name%] calls: - [ pushHandler, [ @applogHandler ] ] seclogHandler: class: %logger.monolog.handlerClass% arguments: [%logger.security.path%, %logger.security.level%] applogHandler: class: %logger.monolog.handlerClass% arguments: [%logger.application.path%, %logger.application.level%] UserMapper: class: %user.mapper.class% arguments: [@logger] services.yml 2/2
• 57.
  // Set UpContainer $container = new ContainerBuilder; $loader = new YamlFileLoader($container, new FileLocator(__DIR__)); $loader->load("services.yml"); // Get Mapper and Logger $logger = $container->get('logger'); 01-basic.php 1/2
• 58.
  Me
• 60.
  5) Event Logging https://github.com/benwaine/ApplicationLoggingTalk/tree/master/ examples/05-StatsDGraphite
• 61.
  Log Entries AsEvents That Occur In Your System
• 64.
  Graphite + StatsD
• 65.
  StatsD • Buckets • Values •Flush
• 66.
  // Creating StatsDClient $connection = new DomniklStatsdConnectionSocket('localhost', 8125); $statsd = new DomniklStatsdClient($connection, "test.namespace"); // simple count $statsd->increment("request");
• 67.
  public function logBusinessEvent($event){ if(!isset($this->events[$event])) { throw new Exception('Invalid Logging Event'); } $this->statsd->increment($this->events[$event]); } public function logBusinessTime($event, $time) { if(!isset($this->events[$event])) { throw new Exception('Invalid Logging Event'); } $this->statsd->timing($this->events[$event], $time); } Logger.php (extract)
• 68.
  while(true) { $number = rand(1, 10); for($x = 0; $x <= $number; $x++) { $user = new User(rand(1, 10000), 'Betty Boo'); $mapper->save($user); $logger->logBusinessEvent(Logger::EVENT_USERCREATED); unset($user); if($x%2 === 0) { // Simulate some kind of security warning when // creating some of the users. $logger->logBusinessEvent(Logger::EVENT_USERWARNING); } } sleep(rand(1, 15)); } 02-example.php (extract)
• 69.
  5) Putting ItAll Together https://github.com/benwaine/ApplicationLoggingTalk/tree/master/ examples/06-AllTogether
• 70.
  Roadmap • Introduction -Why Is Logging Good? • The Mechanics - Code / Examples / Live Demos (GULP) • Logging Strategy - What to do with your logs • Conclusion
• 71.
  Logging Strategy Style 1) Choose a log format and be consistent 2) Make sure common tasks like formatting and filtering are dealt with centrally. 3) Choose a set of rules / style around when to log and be consistent.
• 72.
  Logging Strategy Content 1) Request Start / End (with times and memory) 2) Transition between layers Eg - Control > Model > DB < Model < Control > View 3) Database Queries 4) Business Events 5) Message Queue Activity 6) PHP Errors / PHP Notices
• 73.
  Logging Strategy Volume 1) Request Start / End (with times and memory) Notice 2) Transition between layers Eg - Control > Model > DB < Model < Control > View Info 3) Database Queries Info / Debug 4) Business Events Separate Log 5) Message Queue Activity Debug 6) PHP Errors / PHP Notices Error
• 74.
  Logging Strategy Audience 1) Make it easy to switch log levels (environment variables). 2) Always think of your log consumers when adding log lines.
• 75.
  Grep-able Content Ideas Blameable
• 76.
  Log Aggregation & Presentation
• 77.
  RSYSLOG + LOGSTASH + STATSD + GRAPHITE
• 78.
  RSYSLOG •Send Logs ViaTCP / UDP To A Central Location •Use Filters & Templates To Write Logs To Files / Database Web /var/log/web/%ip-address%/access.log /var/log/web/%ip-address%/error.log LOG /var/log/db/%ip-address%/query.log DB /var/log/db/%ip-address%/slow-query.log
• 79.
  LOGSTASH input { file { path => "/var/log/apache/access.log" type => "apache-access" } } filter { grok { type => "apache-access" pattern => "%{COMBINEDAPACHELOG}" } } output { statsd { # Count one hit every event by response increment => "apache.response.%{response}" } } http://logstash.net/docs/1.1.1/tutorials/metrics-from-logs
• 80.
  LOGSTASH Inputs Filters Outputs 1 amqp 1 checksum 1 amqp 23 nagios 2 eventlog 2 csv 2 boundary 24 nagios_nsca 3 exec 3 date 3 circonus 25 null 4 file 4 dns 4 datadog 26 opentsdb 5 ganglia 5 environment 5 elasticsearch 27 pagerduty 6 gelf 6 gelfify 6 elasticsearch_http 28 pipe 7 generator 7 grep 7 elasticsearch_river 29 redis 8 heroku 8 grok 8 email 30 riak 9 irc 9 grokdiscovery 9 file 31 riemann 10 log4j 10 json 10 ganglia 32 sns 11 pipe 11 multiline 11 gelf 33 statsd 12 redis 12 mutate 12 graphite 34 stdout 13 stdin 13 noop 13 graphtastic 35 tcp 14 stomp 14 split 14 http 36 websocket 15 syslog 15 syslog_pri 15 internal 37 xmpp 16 tcp 16 xml 16 irc 38 zabbix 17 twitter 17 zeromq 17 juggernaut 39 zeromq 18 udp 18 librato 19 xmpp 19 loggly 20 zeromq 20 metriccatcher 21 mongodb 22 stomp
• 81.
  DEMO
• 82.
  Hosted Services
• 86.
  Roadmap • Introduction -Why Is Logging Good? • The Mechanics - Code / Examples / Live Demos (GULP) • Logging Strategy - What to do with your logs • Conclusion
• 89.
  Fin - Questions? Askme to repeat the question. I ALWAYS forget.
• 90.
  Hiring Developers &Testers (Contract / Perm) [email protected]