Drupal Backend Performance and Scalability

Drupal Backend Performance and Scalability

• 1.
  Drupal Backend Performanceand ScalabilityAshok and Christefano
• 2.
  August 7, 2010Aboutme (BTMash)Systems programmer.
• 3.
  Work at CaliforniaInstitute of the Arts.
• 4.
  Working with Drupalsince 2006 (4.6.x)
• 5.
  Primarily help withpatches and upgrades for contrib modules on d.o.
• 6.
  Abuse
• 7.
  Favorites
• 8.
  Flag Content
• 9.
  Fivestar
• 10.
  Nice Menus
• 11.
  Simplenews
• 12.
  Userpoints
• 13.
  Userpoints_badges
• 14.
  User_badge
• 15.
  And more!
• 16.
  Strong interest inserver optimizationAbout me (Christefano)LADrupal Manager
• 17.
  Camp Organizer
• 18.
  Drupal Core Contributor
• 19.
  Person of theYear
• 20.
  D.O and G.D.Owebmaster teams.ResourcesKhalid Baheyeldin
• 21.
  http://2bits.com
• 22.
  Drupal.org infrastructure group
• 23.
  Peter Zaitsev
• 24.
  http://www.mysqlperformanceblog.com/about/GoalsDefine your objectivesand goals first
• 25.
  Do you wantfaster response to the end user per page?
• 26.
  Do you wantto handle more page views?
• 27.
  Do you wantto minimize downtime?
• 28.
  Each is different,though they may be related
• 29.
  Sometimes, there aresome ‘low hanging fruit’ that are easy to see, that provide noticeable improvement with little effort.
• 30.
  Gets hard toachieve more performance (more efforts, low return)
• 31.
  More infrastructure (splittingservers, multiple web/db servers, etc)
• 32.
  Patching Drupal
• 33.
  Revisions to Architecture(CCK, Views)DiagnosisProper diagnosis is essential before proposing and implementing a solution or you are running blind.
• 34.
  Must be basedon proper data.
• 35.
  Analysis of datacollected.
• 36.
  Can lead towardsa few possible paths of optimization.Points of optimizationIntroductionTools to measure and diagnose issuesSpeed Optimizations
• 37.
  On to theServer…Much more complex
• 38.
  Measurement and monitoringtools
• 39.
  Hardware
• 40.
  The LAMP Stack
• 41.
  Linux, Apache, MySQL,PHP
• 42.
  Drupal
• 43.
  Database queries
• 44.
  Modules
• 45.
  CachingValidationAvoid the ‘wildgoose chase’
• 46.
  Validate the resultson a test server
• 47.
  Replicate the siteon a development server (or local machine with the current data)
• 48.
  Backup and Migratemodule help with that
• 49.
  Recreate the site
• 50.
  Gather a timedifference ratio between the test and production server (production server is likely to be much faster)
• 51.
  Give a roughidea on any improvements made.
• 52.
  Measure again andsee if relative times remain the same.The LAMP StackCurrently most commonly used stack for hosting Drupal and similar applications
• 53.
  Linux
• 54.
  Apache
• 55.
  MySQL
• 56.
  PHP
• 57.
  Most of thispart of the presentation applies to *BSD as well.HardwarePhysical server matters
• 58.
  Dedicated
• 59.
  VPS
• 60.
  In the Cloud(Yes, it does work!)
• 61.
  Multiple cores arethe norm
• 62.
  8 > 4> 2 > 1
• 63.
  Lots of RAM(caching the file system and database as much as possible)
• 64.
  Multiple disks
• 65.
  Not really applicableto shared hostingMultiple ServersOne database server, multiple web servers
• 66.
  Can use DNSround robin for load balancing
• 67.
  Or proper loadbalancers
• 68.
  Can use areverse proxy (much like Drupal.org)
• 69.
  Do it onlyif you have the budget
• 70.
  Complexity is expensive(running cost, maintenance time)
• 71.
  Tuning a systemcan avoid (or delay) a splitTesting toolsab/ab2 (Apache Benchmark)
• 72.
  Ab –c 50–n 10000
• 73.
  Do 50 concurrentrequests for up to 10000 requests
• 74.
  Average response timeper second
• 75.
  How many requestshandled per second
• 76.
  Can test withauthenticated sessions
• 77.
  Siege
• 78.
  http://www.joedog.org/index/siege-home
• 79.
  Similar in functionalityto ab
• 80.
  Can test POSTfunctionality
• 81.
  Cannot natively useon windows
• 82.
  JMeter
• 83.
  http://jakarta.apache.org/jmeter/
• 84.
  Desktop application writtenin java (can natively use on windows)Hardware Monitoring toolsTop
• 85.
  Classic Unix/Linux program
• 86.
  Real time monitoring(What is the system doing right now??)
• 87.
  Load average
• 88.
  CPU utilization
• 89.
  Memory usage
• 90.
  List of processes,sorted, with CPU memory
• 91.
  Can change orderof sorting, time interval, grouping, along with many other things.
• 92.
  Htop
• 93.
  http://htop.sourceforge.net/
• 94.
  Similar to topbut for multiple cores
• 95.
  Faster
• 96.
  Colours, stylesHardware Monitoringtoolsatop
• 97.
  http://www.atcomputing.nl/Tools/atop/
• 98.
  Different format andinformation
• 99.
  Shows network statistics
• 100.
  Runs a collectiondaemon in the background
• 101.
  Vmstat
• 102.
  Report memory statistics
• 103.
  Can display inincrements (vmstat 5 means it will show snapshots every 5 seconds)
• 104.
  netstat
• 105.
  Shows active networkconnections
• 106.
  netstat –anp
• 107.
  netstat –anp |grep ESTGraphical Monitoring toolsCacti
• 108.
  http://www.cacti.net/
• 109.
  Available as apackage on Ubuntu, Debian (any other *nix / *bsd?)
• 110.
  Easy to understandgraphs
• 111.
  Displays history overday, week, month, year
• 112.
  Graphs available todisplay statistics for CPU, memory, network, Apache, MySQL
• 113.
  Many others writtenby others available online (can find one for memcached, as an example)
• 114.
  Munin
• 115.
  http://munin.projects.linpro.no/
• 116.
  Very similar toCacti
• 117.
  Can also createyour own monitoring scripts
• 118.
  Nagios
• 119.
  Haven’t used ityet
• 120.
  Supposed to bevery powerful (alerts by email, sms, etc)
• 121.
  Drupal module whichintegrates with toolLinux / BSDUse a proven stable distribution (Debian Stable, Ubuntu LTS, RHEL, CentOS)
• 122.
  Use recent versions
• 123.
  Use whatever yourdistribution your staff has expertise with (most important!)
• 124.
  Try to avoidbloat
• 125.
  Don’t install PostGreSQLif you are only using MySQL, no desktop server, java, etc
• 126.
  Balance compiling ownprograms versus using packages from distro
• 127.
  Compiling provides fullcontrol (and may let you get more out of) on the program.
• 128.
  Can be apain to upgrade.
• 129.
  Using packages onlylet you use whatever the latest version may be (APC was not available for RHEL until middle of last year)
• 130.
  Upgrades are relativelypainless.ApacheMost popular, supported, and feature rich
• 131.
  Fairly stable
• 132.
  Can also beenabled with too many unnecessary modules
• 133.
  Disable unnecessary modules(mod_proxy, cgi_module, etc may not be necessary on server if not hosting sites which take advantage of it)
• 134.
  Smaller process =more users can access site
• 135.
  ‘apachectl –M’ displaysall modules currently enabled on site.
• 136.
  Monitoring Tool –apachetop
• 137.
  http://www.webta.org/projects/apachetop/
• 138.
  Reads and analysesApache’s access log
• 139.
  Show all /recent hits
• 140.
  Requests / s,KB / s, KB / request
• 141.
  Good to detectcrawlers
• 142.
  Run using ‘apachetop–f /path/to/site/access.log’Apache OptimizationsMaxClients (prevent swapping / thrashing)
• 143.
  Too low –cannot server enough clients
• 144.
  Too high –you run out of memory, and you start swapping (server dies and cannot server ANY clients)
• 145.
  MaxRequestsPerChild
• 146.
  Tune it toterminate the process faster and free up memory.
• 147.
  KeepAlive
• 148.
  Keep it on.
• 149.
  mod_gzip/deflate
• 150.
  Compress css, js,etcAlternativeslighttpd
• 151.
  http://www.lighttpd.net/
• 152.
  Popular with Rubyon Rails
• 153.
  Lighter than Apache(can go as low as 1MB)
• 154.
  Known to haveleaks
• 155.
  Nginx
• 156.
  http://nginx.net/
• 157.
  More stable thanlighty
• 158.
  Easier to setupthan lighty
• 159.
  Drupalcampla is runningon it…
• 160.
  Both run PHPas only as FastCGI
• 161.
  Separate processesAlternatives (cont’d)Varnish
• 162.
  HTTP accelerator
• 163.
  Set it upas a reverse proxy to send the call to apache if it cannot (or should not) serve something itself
• 164.
  Serve anonymous pagerequests and static files
• 165.
  Not particularly difficultto set up.
• 166.
  Does require somelevel of tuning.
• 167.
  Used on Drupal.org
• 168.
  Will be usedon ladrupal.org
• 169.
  Squid
• 170.
  Similar to Varnish
• 171.
  Precursor to Varnish(so go with the above)MySQLMost popular database for Drupal
• 172.
  Not necessarily thebest database from technology standpoint (default ending lacks ACID, transactions, concurrency) but still does a good job.
• 173.
  Easy to setup
• 174.
  Various pluggable engines(InnoDB, Archive, etc)MySQL Monitoringmtop / mytop
• 175.
  Like top, but for MySQL
• 176.
  Real time monitoring (no history)
• 177.
  Shows slow queries and locks
• 178.
  If you have neither –
• 179.
  SHOW FULL PROCESS LIST – mysqladmin processlist
• 180.
  Mysqlreport
• 181.
  http://hackmysql.com/mysqlreport
• 182.
  Reports on server– no recommendations (documentation on site explains a lot, however)
• 183.
  MySQL DB Tuner
• 184.
  http://wiki.mysqltuner.com/MySQLTuner
• 185.
  Another one athttp://www.day32.com/MySQL/
• 186.
  Provides fairly decentrecommendations (and warnings) on MySQL settingsMySQL MonitoringSlow Query Log
• 187.
  Can be enabledin your my.cnf file
• 188.
  Lists queries morethan N seconds
• 189.
  Can also listqueries which do not have indexes (if you choose)
• 190.
  Very useful toidentify potential bottlenecks
• 191.
  Helped me identifythat the COUNT(*) query used through the search module was cause for site search crawl
• 192.
  Removed count witha base query which cut the search query and render to less than half the time
• 193.
  Best way tointerpret all of it
• 194.
  mysql_slow_log_parser_script(http://www.mysqlperformanceblog.com/files/utils/mysql_slow_log_parser)MySQL EnginesMyISAM
• 195.
  Fast reads
• 196.
  Less overhead
• 197.
  Poor concurrency (tablelevel locking)
• 198.
  InnoDB
• 199.
  Transactional
• 200.
  Slower in somecases (SELECT COUNT(…))
• 201.
  Lots of settingsto analyze and change
• 202.
  Better concurrency (goodfor tables that are constantly hit, such as sessions, watchdog, etc)
• 203.
  New Engines indevelopment
• 204.
  Falcon (promising ACID)
• 205.
  SolidDB
• 206.
  Concerns over themany new variants of MySQL
• 207.
  MariaDB, Percona (forkedInnoDB), Google Patchsets, Mysql 5.5MySQL TuningQuery Cache, Key Buffer
• 208.
  VERY Important!
• 209.
  Also important
• 210.
  Table Cache (cachesyour tables)
• 211.
  sort_buffer_size, read_buffer_size
• 212.
  tmp_table_size (required tobe the same or greater than query cache)
• 213.
  InnoDB
• 214.
  Innodb_buffer_pool_size (most importantparameter)
• 215.
  Set this valueto be 50 – 80% of total memory you wish to allocate towards MySQL
• 216.
  Loads of settings!
• 217.
  http://www.mysqlperformanceblog.com/ is agreat resource to learn lots about tuning MySQLMySQL ReplicationUsed on Drupal.org
• 218.
  INSERT/DELETE/UPDATE queries goto master
• 219.
  SELECT queries goto slave(s)
• 220.
  Provide noticeable improvement
• 221.
  Usage on http://www.zimmertwins.comdramatically sped up the site
• 222.
  D.O has alsoseen noticeable improvements
• 223.
  Patch for 6.xseen at http://drupal.org/node/147160
• 224.
  Beware of complexity(connection between master/slave goes wrong, can bring down site)
• 225.
  Did extensive tuningof Zimmer Twins DB server
• 226.
  Noticeable improvement despitelack of querying slave server
• 227.
  Removed need forslave server.PHPUse a recent, stable release
• 228.
  Drupal 7 willrequire 5.2.x, as do a few current 6.x contributed modules
• 229.
  Install an Op-codecacher / accelerator
• 230.
  eAccelerator
• 231.
  APC
• 232.
  Xcache
• 233.
  Zend optimizer (commercial)
• 234.
  Very useful inbringing down memory usage for a siteOp-code CachingBenefits
• 235.
  Lowered memory usage
• 236.
  Significant decrease inCPU utilization
• 237.
  Can be biggestimpact on a site
• 238.
  Usage on http://www.zimmertwins.comlowered memory usage per process from 20 MB down to less than 4 MB
• 239.
  Usage on http://calarts.edulowered memory usage per process down from 35 MB down to 7 MB
• 240.
  Cached page usagewas even lower (2 MB or less)
• 241.
  2bits.com has writtencomparisons between the 3 cachers at http://tinyurl.com/5uo874
• 242.
  Drawbacks
• 243.
  May crash
• 244.
  May require restartsafter updating code
• 245.
  Using the capistranoscript by the TxT folk (who use xcache) may be useful.Op-code CachingOp-code caching will not work in all circumstances
• 246.
  Network connections
• 247.
  Sorting arrays
• 248.
  Database queries
• 249.
  All of theabove
• 250.
  Some culprit modules
• 251.
  Tagadelic
• 252.
  Node access modules
• 253.
  Admin menu
• 254.
  Forum
• 255.
  TrackerRunning PHPmod_php
• 256.
  Standard php moduleused by Apache
• 257.
  No state retainedbetween requests
• 258.
  Few issues
• 259.
  Well tested andsupported
• 260.
  May not needto use anything else
• 261.
  Can be aslow as 10 – 12 MB
• 262.
  Can easily beover 100 MB (depending on drupal modules installed, apache modules enabled on server, etc)
• 263.
  mod_cgi
• 264.
  Old way ofrunning php
• 265.
  Don’t use itnow (inefficient, each request forks process)
• 266.
  Fast CGI
• 267.
  Much faster thancgi
• 268.
  Can be usedwith nginx, lighttpd, apache
• 269.
  Apache + fcgitested to be more stable with lower memory usage than mod_phpDrupalMainly database intensive
• 270.
  Can be aCPU hog
• 271.
  Can be memoryintensive
• 272.
  Some modules areknown to be slow
• 273.
  Site may notbe affected by bottleneck.
• 274.
  Maintenance tips
• 275.
  Disable unnecessary modules
• 276.
  Make sure cronruns regularlyDrupal ToolsDevel
• 277.
  http://drupal.org/project/devel
• 278.
  Total page execution
• 279.
  Query execution time
• 280.
  Query log
• 281.
  Memory utilization
• 282.
  Can be combinedwith stress testing
• 283.
  Log pages whichuse most queries
• 284.
  Log pages whichtake most time to generate
• 285.
  Trace
• 286.
  http://drupal.org/project/trace
• 287.
  Use for debugging
• 288.
  Traces output, hookinvocations, warnings
• 289.
  Can filter byquery typeModule calls over networkDoes your module do interesting things over the network?
• 290.
  Email users (og)
• 291.
  Call web widgets/ APIs (digg, youtube, flickr)
• 292.
  Cache data asmuch as possible (from web widgets/apis)
• 293.
  Use helper modulesto aid with reducing bog
• 294.
  job_queue module
• 295.
  Queue Mail module
• 296.
  SMTP mail module
• 297.
  WeatherDrupal page cachingPageCaching
• 298.
  Primarily for anonymoususers
• 299.
  Doesn’t necessarily affectauthenticated users
• 300.
  May expire toosoon
• 301.
  Set minimum cacheexpiry time
• 302.
  Aggressive cache isnot compatible with all modules (statistics from core) but can provide better performance.
• 303.
  Other types ofcaching are not necessarily turned off
• 304.
  Filter
• 305.
  Menu
• 306.
  Blocks (6.x)
• 307.
  Variables
• 308.
  FormsUseful Drupal cachingmodulesBoost
• 309.
  http://drupal.org/project/boost
• 310.
  Creates HTML forpages and stores it in files
• 311.
  Requires changes to.htaccess files and symlinks (admin section of module now shows what to write into htaccess file)
• 312.
  Usable on sharedhosts as well as VPS and dedicated servers
• 313.
  Enhances ability tohandle traffic spikes (from anonymous users)
• 314.
  Can also usemodule to display site while in maintenance mode (only uncached pages and authenticated users will get maintenance mode page)
• 315.
  Development Seed hascase study on using boost for Anti Poverty event site that notes performance gain by nearly 1000 times (http://tinyurl.com/l7ktfy)
• 316.
  Cache Router
• 317.
  FS Fastpath
• 318.
  Memcached
• 319.
  Authenticated User CachePluggablecachingUse $conf variable in settings.php
• 320.
  $conf[‘cache_inc’] = ‘./path/to/caching_mechanism.inc’;
• 321.
  Allows you tohave a custom caching module
• 322.
  Can even usethis to completely disable caching (DO NOT USE ON A PRODUCTION SITE)MemcachedDistributed object caching in memory
• 323.
  Written by dangafor livejournal
• 324.
  Lives in memory
• 325.
  Can span multipleservers (
• 326.
  Drupal install usespluggable caching mechanism
• 327.
  Should be seamlessfor Drupal 6.x
• 328.
  Drupal 5.x requiresa patch + schema changes
• 329.
  Can lower dbqueries in ½
• 330.
  Has own requirements(apache needs to restarted, more instances = more complexity)Changing the search mechanismDrupal Core Search is good…?
• 331.
  But can quicklyget bad with more and more content
• 332.
  DB Tuning canonly go so far
• 333.
  With nearly 1Mnodes, search on zimmertwins.ca would take over 30 seconds.
• 334.
  Try another engine!
• 335.
  ApacheSOLR
• 336.
  Very fast
• 337.
  Pain to configure
• 338.
  Requires server onwhich Java can be installed
• 339.
  Available as adrupal module
• 340.
  Acquia offers serviceto use their setup (wholly takes load off your server)
• 341.
  LuceneAPI
• 342.
  Not as fastas ApacheSolr, but easy to install and can work with that before moving on to Solr
• 343.
  Google CSE modulemight also be a good fitOther optionsUsing an optimized distribution
• 344.
  Pressflow
• 345.
  http://fourkitchens.com/pressflow-makes-drupal-scale
• 346.
  Supports database replication
• 347.
  Only supports MySQL
• 348.
  Cleanly supports reverseproxies such as Varnish
• 349.
  Optimized for PHP5
• 350.
  Mercury
• 351.
  Build based onPressflow
• 352.
  Is an AmazonMachine Image to be used as instance of Amazon EC2
• 353.
  Install image containsinstallation of Solr, Varnish, Memcached
• 354.
  Tuned Apache settings
• 355.
  Performance is verypromising.Other options (cont’d)Patching Drupal
• 356.
  Other name forit is ‘hack core’
• 357.
  Not recommended (unlessyou know what you are doing)
• 358.
  Sometimes it isnecessary
• 359.
  Hack core froma safe distance if possible
• 360.
  Copy core moduleinto sites/site_name_directory/modules and apply changes
• 361.
  Use pluggable cachingto have your own version of caching
• 362.
  Create schema changesin a separate module (so you can remove them cleanly in the future)
• 363.
  Create own moduleand override original mechanism if process is called through form builder (change the submission or validation to point to yours)
• 364.
  Have an easyway to track changes you have madeOther options (cont’d)Case 1:
• 365.
  User login onzimmertwins.ca was painfully slow (5+ seconds per user)
• 366.
  Gist of problem:DB not using index on username due to lower()
• 367.
  Original issue athttp://drupal.org/node/83738 (now ongoing in a new issue)
• 368.
  ‘Bug’ has beenaround since 2006
• 369.
  Solution: Modified patchon d.o for site
• 370.
  Copied user moduleinto sites/all/modules to use override code
• 371.
  user login timenow less than 0.1 seconds
• 372.
  As a note,d.o has also been patched (login has required matching case for past few months)
• 373.
  Case 2:
• 374.
  Comments do nothave index on user id: problem with abuse module
• 375.
  Solution: Created indexon user id as an update
• 376.
  Added update tomodule that requires this schema change (my own module)
• 377.
  Loading for commentby user no longer an issue.Module DevelopersTake advantage of caching
• 378.
  Use memory wisely
• 379.
  Unset the variableif you do not have a need for it down the line)
• 380.
  Save variable tomemory for future use so processing is not done multiple times
• 381.
  Take Advantage ofAHAH functionality
• 382.
  Fewer queries
• 383.
  Not reloading thepage
• 384.
  Saving bandwidth
• 385.
  Learn to usejQuery (falls in line with above)(Possibly) Related Camp PresentationsDrupal Performance: Flash Storage as a Game Changer
• 386.
  Christoph Weber
• 387.
  Using Flash Storageto power your database for one.
• 388.
  Install Drupal 7with Ubuntu 9.10 on Amazon EC2
• 389.
  HarnishGoradia (thatlaguy)
• 390.
  Building on SOLRSearch
• 391.
  John FialaThank you!Questions?Answers? Tips from you all? Discuss!