Evented applications with RabbitMQ and CakePHP
- 1. Evented Applications with RabbitMQ , Python, and CakePHP
- 2. What the wibblefish? All about offloading work, and doing things in ‘realtime’ Make applications that complete tasks in a distributed/scalable fashion. Doing work outside the request/response cycle of a UI/API.
- 3. Work to offload Noti cations (SMS, Email, Postal mail). Logging. Import & export. Process image & video. Reporting. Webhooks.
- 4. Isn’t cron good enough? Cron tabs are great with one server. Cron doesn’t scale to multiple machines. Cron doesn’t solve the whole problem space.
- 5. Problems with Cron Isn’t suitable for ‘real-time’ results. Requires a set schedule. You might spin up processes that do nothing. Or you might not have enough capacity. Parallelizing is a pain.
- 6. Queues to the rescue! Queues accept and dispatch events. Events can contain almost anything*. Events can be roughly equal to ‘jobs’. A single event can trigger lots of downstream work.
- 7. Architectural benefits Separation of concerns. Collection of mini/small applications. No monolithic applications. Decoupled, and easier to test. Easy to scale. Busy service = more of them.
- 9. RabbitMQ Written in Erlang. Open source and Free. Really Fast. Socket protocol that is has cheap connections. Data persists after a crash.
- 10. Queue basics Producers create events. Consumers eat messages, and do work. messages == events.
- 13. Queue Basics Producer Consumer
- 14. Queue Basics Consumer Producer Consumer
- 15. Queue Basics Consumer Producer Consumer Consumer
- 16. Exchanges Exchanges abstract queues from the producer. Route messages to queues, that are bound by routing keys. Allows consumers to listen to speci c routing keys. Multiple consumers can listen to the same events.
- 19. Exchanges page_view Consumer Producer
- 20. Exchanges page_view Consumer import Consumer Producer
- 21. Exchanges page_view Consumer import Consumer Producer Consumer error, page_view
- 22. Topic subscription Consumers can also listen to topics. Which are keys separated by dots. e.g page_view.search, page_view.update Consumer could listen to page_view.*
- 23. Producing Can create messages with many tools. PHP, python, node, ruby, etc. Messages can contain any stringy data (JSON)
- 24. Data for messages Only string-y things can be stored in messages. JSON encoded data in message. Key to value stored in Redis/Memcache. Keep messages small. RabbitMQ performs better with small messages.
- 25. Consuming Use a daemon process. Daemons in PHP can be done, but they kinda suck. Usually use sleep(2) or other workaround. sleep(2) moves you away from ‘real time’.
- 26. Aw bugger..
- 27. Use a better tool Other tools are much better for making daemons. Python (pika, kombu, sparkplug) NodeJS (node-amqp) Ruby (carrot, ruby-amqp)
- 28. Managing mixed applications Many ways to get it done. Avoid duplicating code. Thin daemons + worker processes.
- 29. Thin daemons Python/node daemon accepts messages. Delegate to PHP worker processes. Keeps PHP’s fork + die work ow. Re-use application code by calling shells.
- 30. Thin daemons PHP
- 31. Thin daemons PHP
- 32. Thin daemons Py PHP
- 33. Thin daemons Py PHP Py
- 34. Thin daemons Py PHP Py Py
- 35. Thin daemons Py PHP PHP Py Py
- 36. Thin daemons Py PHP PHP Py PHP Py
- 37. Thin daemons Py PHP PHP Py PHP Py PHP
- 38. Demo Time
- 39. Overview Use CakeLog to re events. Use CakePHP shell’s for actual work. Thin python daemon for processing messages.
- 40. Python + sparkplug Python module for creating queue listeners. Created by a developer at FreshBooks. Ini con guration le de nes behaviour.
- 41. What we’ll try: User events logged during application. Log events to the database.
- 42. Insert code
- 43. Limitations of using shells Daemon and shell processes have to be on the same box. This can be limiting as you scale, depends on how you structure applications.
- 44. Solutions Use HTTP. Have thin daemons dispatch back to the front- end application using HTTP. Use Basic/Digest authentication. Use checksums.
- 45. HTTP daemons PHP
- 46. HTTP daemons PHP
- 47. HTTP daemons PHP Py
- 48. HTTP daemons Web PHP Py
- 49. HTTP daemons Web PHP Py Web
- 50. HTTP daemons Web PHP Py Web Web