Celery in the Django
Download as PPTX, PDF4 likes1,927 views
This document discusses Celery, an asynchronous task queue/job queue based on distributed message passing. It describes Celery as a distributed, asynchronous task queue. It provides examples of using Celery for tasks like sending thousands of emails or handling computationally heavy queries. The document outlines some benefits of Celery like making tasks distributed and allowing configuration of queues, routing, and task utilities. It also discusses some pitfalls encountered with Celery and complaints about it.
More Related Content
![[Quase] Tudo que você precisa saber sobre tarefas assíncronas](https://cdn.slidesharecdn.com/ss_thumbnails/tarefasasincronas-170505123700-thumbnail.jpg?width=560&fit=bounds)
Celery in the Django
- 1. 叢林裡的小芹菜 Celery in the Django By Walter Liu
- 2. About me Learn Python since 2004 Was a Chief Technical Director in Softstar. Now an Architect in Trend Micro Light Celery user for several months.
- 3. Celery is X Distributed Asynchronous Task Queue
- 4. Have you met? Deliver thousands to millions e-mails to your members? Many computation heavy queries at the same time that blocks your web site or DB? Query slow or too many connection to external API? (like Twitter/Facebook) Long run tasks that occupied your web server resources?
- 6. Why Celery? POC without Celery Add a “@celery.task” and the tasks become distributed tasks. Multi-queue, routing, all configurable. Many utility functionality to help task execution Group/Chord/Chain/Callback Retry/timeout Etc…..
- 7. from celery import Celery @celery.task( max_retries = 3, default_retry_delay=2 * 60, # retry in 2 minutes. rate_limit = 20, time_limit = 10, # hard time limit soft_time_limit = 5, ignore_result = True, acks_late = True, # make sure this task will be done. ) def send_twitter_status(oauth, tweet): try: twitter = Twitter(oauth) twitter.update_status(tweet) except (Twitter.FailWhaleError, Twitter.LoginError), exc: raise send_twitter_status.retry(exc=exc)
- 8. Understand the personality of your celery What happen if the program throw an exception? What happen if the program hard/soft time limit? What happen if the hard/soft time limit reached in your group/chord function? Many others.
- 9. Our pitfall/trouble cases Many, too many small tasks For example 1 billion per day. Large task argument (workaround => store task at cache) Send file directly Better not wait another task in a task. Using Redis as AMQP and no HA. Hard time limit reached in Chord function. Multi-process and multi-thread problem in Windows Logging. Queue management
- 10. Some internet complain about Celery Only Python API change and break his program 2.5.x => 3.0.x Pylint
- 11. Some similar things Gearman Huey Advanced Python Scheduler
- 12. Recruiting – we’re using Front-end: not very limited. Backend Python Django MongoDB Redis/Memcache Celery