Async task with celery

A quick guide on how to use celery to accomplish async task in application.

How it works for celery-based async tasks

  • Decorated function as celery task
  • Broker (e.g. rabbitmq, redis, and etc.) to queue up the task
  • Celery worker on the background to actually execute the task

Installation

# install a broker
$ sudo apt-get install redis-server [or rabbitmq-server]
$ sudo /etc/init.d/redis-server start
or $ sudo /etc/init.d/rabbitmq-server start

# install celery
$ sudo pip install celery
$ sudo pip install redis [if using redis as broker]

Define a task

# celery_tasks.py
from celery import Celery

celery_app = Celery('celery_tasks', broker='redis://localhost:6379/0')

@celery_app.task
def sum_to_one_million():
    return sum(range(1000000))

Start worker

$ cd ~/workspace/celery_project
$ celery -A celery_tasks worker --loglevel=info

Invoke the task

  • in interactive shell 
    >> from celery_tasks import sum_to_one_million
>> print(sum_to_one_million())
  • in a flask app
from flask import Flask
from celery_tasks import sum_to_one_million

app = Flask(__name__)

@app.route("/calcSync")
def calc_sum_sync():
    return sum_to_one_million()

@app.route("/calcAsync")
def calc_sum_async():
    sum_to_one_million.delay()
    return "Calculation started in the background."

References

celery python