Queue Everything and Please Everyone
3 likes2,139 views
Vaidik Kapoor presented on using message queues and task queues to improve the responsiveness of web applications. He discussed how queues can be used to offload processing tasks from the main request-response cycle to improve user experience. Some key points included: - Message queues like RabbitMQ and Redis with libraries like HotQueue and PyRes can be used to asynchronously process tasks like data/media processing and cache updates. - Celery is a popular task queue library that uses RabbitMQ and supports many frameworks. It allows distributing and managing cron-like tasks reliably. - Proper integration of task processing with user interfaces is important using techniques like notifications, flashes, and polling to update users.
More Related Content
Queue Everything and Please Everyone
- 1. Queue Everything & Please Everyone *** Vaidik Kapoor
- 2. Who am I? ➔ Undergraduate student. ➔ Involved with: ✔ Mozilla as a Contributor and Rep ✔ Drupal as a Contributor ➔ Twitter: @vaidikkapoor ➔ Github: vaidikkp ➔ Web: vaidikkapoor.info
- 5. Request Response Client Task 1 Task 2 Task 3 Task N Why do everything at once?
- 6. Break the Request-Response Cycle Request Response Client Task 1 Task N Task 2 Task 3
- 7. Meet Queues
- 8. What can Queues help with? ➔ Background Processing ✔ Data Processing ✔ Media Processing ✔ Updating Caches ➔ Anything that you want to offload off your server. ➔ Improve the overall User Experience
- 9. Redis + HotQueue / PyRes
- 10. Redis + HotQueue PRODUCER: from hotqueue import HotQueue queue = HotQueue("myqueue", host="localhost", port=6379, db=0) queue.put(message) CONSUMER / WORKER: from hotqueue import HotQueue queue = HotQueue("myqueue", host="localhost", port=6379, db=0) while True: message = queue.get() # do something awesome
- 11. Redis + PyRes ➔ Python clone of Github's Resque ➔ Offers a lot more features than HotQueue ✔ Job failure handling ✔ Queue status information ➔ Comes with Monitoring System written in Ruby
- 12. Redis + PyRes PRODUCER: from pyres import ResQ r = Resq() class WebPage: queue = “screenies” @staticmethod def perform(urll): # save the screenshot r.enqueue(WebPage, 'http://python.org') CONSUMER: ./pyres_worker screenies
- 13. Message Queue A system for enabling asynchronous processing of discrete tasks.
- 14. What can MQs help with? ➔ Everything that generic Queues can help with. ➔ Increase Reliability ➔ Cron Jobs (Celery)
- 15. Available MQ Solutions ➔ RabbitMQ ➔ Amazon Simple Queue ➔ Apache MQ ➔ Gearman ➔ Starling ➔ OpenAMQ ➔ Sun Java Message Queue System
- 16. Message Queue Protocols ➔ AMQP: Advanced Message Queue Protocol ➔ JMS: Java Messaging Service ➔ STOMP: Streaming Text Oriented Messaging Protocol
- 17. Criteria for Broker Selection ➔ Difficulty in Recovery ➔ Relatively low level of required maintenance ➔ Ease of Deployment ➔ Durability ➔ Persistence ➔ Community Support ➔ Cluster Support ➔ What language is it written in?
- 18. RabbitMQ ➔ A Message Broker ➔ Implements AMQP
- 19. Pika & Other AMQP Libraries
- 20. Celery ➔ An amazing Task Manager ➔ Batteries Included ➔ Uses RabbitMQ as broker (or almost anything) ➔ Libraries for most of the common web frameworks like Django, Flask, Pyramid ➔ Supports multiprocessing. ➔ Distribute tasks easily. ➔ Makes life easy
- 21. Celery Example from celery.task import task @task(): def take_screenshot(url): # magical code to take screenshot comes here
- 22. How to put it all together?
- 23. The General Setup Task Web App Broker Manager Worker Database Server (s)
- 24. Then what?
- 27. Flask's Flashes
- 28. Poll using AJAX
- 29. WebSockets
- 30. Email
- 32. Isolate Isolate code, make reusable components.
- 34. Unit Testing?
- 35. Review ➔ Choose a system according to needs ➔ Build a robust system ➔ Integrate seamlessly with your UI ➔ Isolate & Recycle: make it a habit ➔ Improve the overall UX! ➔ Please everyone, even yourself!
- 36. References ➔ PyRes - http://pyres.readthedocs.org/ ➔ RabbitMQ - http://www.rabbitmq.com/ ➔ Celery - http://celeryproject.org/ ➔ Amazing Article - http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
- 37. Thank You!