Building Distributed System with Celery on Docker Swarm

Building Distributed System with Celery on Docker Swarm

• 1.
  Building Distributed System withCelery on Docker Swarm Wei Lin 2016/06/03
• 2.
  2 About Me • WeiLin • @Wei_1144 • [email protected] • Mostly worked in the fields of: – marketing – strategy-planning – project-management
• 3.
  3 Parallel computing https://resin.io/blog/what-would-you-do-with-a-120-raspberry-pi-cluster/
• 4.
  4 Monitor and Control
• 5.
  5 Distributed system
• 6.
  6 Distributed system (II)
• 7.
  7 Neural network h1 hh2h3
• 8.
  8 Communicate via Celery
• 9.
  Celery
• 10.
  10 Celery flowchart
• 11.
  Send message toa specific queue
• 12.
  12 Send message http://docs.celeryproject.org/en/latest/userguide/calling.html kick.apply_async([‘neuron_x’]) kick.apply_async([‘neuron_x’], routing_key= ‘neuron_h1’)
• 13.
  13 Configuration file
• 14.
  14 Task Message Example http://docs.celeryproject.org/en/latest/internals/protocol.html
• 15.
  15 Routing Tasks http://docs.celeryproject.org/en/latest/userguide/routing.html
• 16.
  Assign an uniqueworker for a specific queue
• 17.
  17 Starting the Workerhttp://docs.celeryproject.org/en/latest/userguide/workers.html https://github.com/celery/celery/issues/3065 $celery -A IoT worker -n neuron_h1 -Q neuron_h1 --concurrency=1
• 18.
  Docker
• 19.
  19 Host and containers https://denibertovic.com/talks/supercharge-development-env-using-docker/#/10
• 20.
  20 Run worker container
• 21.
  21 Deploy containers
• 22.
  22 Celery worker containers asneurons h h1 h2 h3 kick.apply_async( routing_key = ‘neuron_h1’) getData.apply_async(routing_key = ‘neuron_h1’) .get()
• 23.
  23 “Celery + Docker-Swarm” So?
• 24.
  I think…
• 25.
  25 Docker file/image/container https://denibertovic.com/talks/supercharge-development-env-using-docker/#/11
• 26.
  26 Think OO • “From”to Inherit • Dockerfile to Encapsulate • Docker Image as Class • Docker Container as Object
• 27.
  27 Docker Swarm &Celery doSomething.apply_async (routing_key = ‘host_NY’) Taipei Tokyo New York getData.apply_async(routing_key = ‘neuron_h1’) .get()
• 28.
  28 Monitor and Control
• 29.
  29 OOAD Station C StationB Station A wait.apply_async ([xxx], routing_key = ‘station_B’) wait.apply_async ([xxx], routing_key = ‘station_A’)
• 30.
  30 Bound services
• 31.
  31 Effects • easier toconvert a system into its distributed version. – Decoupling, DI/IC (dependency injection / inversion of control ) – Distributed = Shared – Load-Balancing doSomething.apply_async(routing_key = ‘host@remote’) getData.apply_async(routing_key = ‘neuron_h1’) .get()
• 32.
  32 Load-Balancing controller model database addItem.apply_async([order], routing_key = ‘model’) addItem.apply_async ([‘order’], routing_key = ‘database’)
• 33.
  33 Parallel computing https://resin.io/blog/what-would-you-do-with-a-120-raspberry-pi-cluster/
• 34.
  34 Deploy parallel computing •Deploy containers over Docker Swarm – Scale up to 480 containers • docker-compose up –d • docker-compose scale celery=480 • Send task messages to worker containers results = [ doSomething.apply_async([data]) for data in dataset ] • But, needs root password-less SSH
• 35.
  35 SETI@home
• 36.
  36 One for all,All for one https://github.com/Wei1234c/OneForAll_AllForOne/blob/master/celery_projects/OneForAll_AllForOne.md • run a Docker image to join distributed parallel computing cluster, that’s all. docker run –d wei1234c/musketeer (for example) • Works on RPi and amd64 PC One for all, All for one
• 37.
  37 Summary • Caution: – Distributedsystems theory for the distributed systems engineer • Easy to use doSomething.apply_async ([xxx], routing_key = ‘host_NY’) getData.apply_async(routing_key = ‘neuron_h1’) .get() • Canvas – Chain – Group
• 38.
  Q & A
• 39.
  39 References References : • Distributedsystems theory for the distributed systems engineer • Celery user guide • Docker document • MQTT Message Type and Flows • Celery on Docker Swarm • IoT as Brain • One for all, All for one