Does Django scale? An introduction to Scalability

Does Django scale? An introduction to Scalability

• 1.
  “Does Django scale?” Anintroduction to Scalability Python Barcelona David Arcos
• 2.
  Hi! I’m David ● I’ma backend developer specialized in distributed systems, scalability and security ● >10 years building SaaS platforms in different sectors ● CTO at Lead Ratings – SaaS solution that uses Machine-Learning algorithms to predict business metrics (conversion, billing, churn...) ● Associate professor at ESADE – Teaching “Cloud Computing” ● PyBCN meetup assistant since 2008, organizer since 2013
• 3.
  1) Does DjangoScale? Questions
• 4.
  https://stackoverflow.com/questions/886221/does-django-scale
• 5.
  https://docs.djangoproject.com/en/dev/faq/general/#does-django-scale
• 6.
  1) Does DjangoScale? YES 2) What is Scalability? Questions
• 7.
  Scalability “The potential ofa system to be enlarged to accommodate a growing amount of work”
• 8.
  “A Word onScalability” (2006) “A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added” - Werner Vogels (Amazon CTO) https://www.allthingsdistributed.com/2006/03/a_word_on_scalability.html
• 9.
  Will it scale? Analgorithm/protocol/system does scale, if it is suitably efficient and practical when applied to large situations. In example: – a large input data set – a large number of outputs or users – a large number of participating nodes If it fails when increasing, it doesn’t scale.
• 10.
  Performance != Scalability ● Performanceis a quality metric: – “The time it takes to execute one request” ● Scalability: – The ability to maintain that performance, under increasing load – Or: increase performance when adding resources
• 11.
  Where is thestate? ● Scaling is much easier if: – The app servers are stateless – The database keeps the state ● So: – Vertical scalability on (stateful) db servers – Horizontal scalability on (stateless) app servers
• 12.
  Vertical / horizontalscaling
• 13.
  Vertical scaling ● Scale up/down:add/remove resources to a single node – Adding more power (faster CPU, more CPUs, more memory, a bigger disk…) to an existing machine ● Very common on databases (stateful systems) ● Easier… when possible
• 14.
  Vertical scaling… but... ● But: –There are hardware limits ● RDS current limit of 488 GiB of memory (db.r4.16xlarge) – Diminishing returns ● At some point, the level of benefits gained is less than the amount of money invested. – Involves downtime ● Keep a hot spare! ● RDS Multi A-Z has a “standby” replica with automatic failover
• 15.
  Horizontal scaling ● Scale out/in:add/remove nodes to a system – Adding new computers to a cluster ● Very common on (stateless) web servers ● High availability – doesn’t matter if some nodes crash ● Cheaper (HW is cheap, VMs are cheap)
• 16.
  Horizontal scaling.. but... ● But: –Architecture must support it ● Keep the state in external db! – Management overhead ● Automate it! ● Sysadmin operations must be treated as software + config
• 17.
  1) Does DjangoScale? YES 2) What is Scalability? 3) What is Django? Questions
• 18.
  “The web frameworkfor perfectionists with deadlines” https://www.djangoproject.com
• 19.
  Who uses Django? ● Instagram ● Pinterest ● Spotify ● Disqus ● Bitbucket ● MozillaFirefox ● Dropbox ● Open Stack ● NASA ● The Washington Post ● National Geographic ● The Onion ● Eventbrite ● Prezi ● Reddit Gifts ● Open Knowledge Foundation ● MacArthur Foundation
• 22.
  DRF: Django RESTframework https://www.django-rest-framework.org “A powerful and flexible toolkit for building Web APIs”
• 23.
  http://www.celeryproject.org Celery: Distributed TaskQueue
• 24.
  Other Python webframeworks
• 25.
  1) Does DjangoScale? YES 2) What is Scalability? 3) What is Django? 4) How to scale Django? Questions
• 26.
  How to ScaleDjango? ● Measure first, find bottlenecks – Rinse and repeat ● Focus on database performance – Indexes, queries, replicas... ● Cache all the things! – Excellent cache documentation ● Run slow tasks in async workers – (when possible)
• 27.
  “High Performance Django”book https://highperformancedjango.com
• 28.
  1) Does DjangoScale? YES 2) What is Scalability? 3) What is Django? 4) How to scale Django? Questions
• 29.
  Thanks for attending! AtLead Ratings we are looking for: – Python Developers – Data Engineers – Data Scientists
• 30.
  Questions?