Huey as crontab alternative in python

Huey is a little task queue that supports:

  • multi-process, multi-thread or greenlet task execution models
  • schedule tasks to execute at a given time, or after a given delay
  • schedule recurring tasks, like a crontab
  • automatically retry tasks that fail
  • task result storage
  • consumer publishes event stream, allowing high-fidelity monitoring

How to use:

Installation

# install redis acts as queue for task storage in huey
$ sudo apt-get install redis-server
$ sudo /etc/init.d/redis-server start

# install huey
$ pip install huey

# [optional] install greenlet if for IO-bound tasks
$ pip install greenlet

Create huey config

# config.py
from huey import RedisHuey

huey = RedisHuey('test', host='localhost')

Create tasks

# tasks.py
from config import huey
from huey import crontab
from datetime import datetime

@huey.periodic_task(crontab(minute="*/1"))
def print_hello():
    print("hello", datetime.now())


@huey.periodic_task(crontab(minute="*/1"))
def print_world():
    print("world", datetime.now())

@huey.periodic_task(crontab(minute="*/1"))
def print_ok():
    print("ok", datetime.now())

Create main executor

from config import huey
from tasks import print_hello, print_world, print_ok


if __name__ == "__main__":
    print_hello()
    print_world()
    print_ok()

Run scheduler

# with process for cpu-intensive tasks
$ huey_consumer.py main.huey -k process -w 4

# with gevent for io-bound tasks
$ huey_consumer.py main.huey -k greenlet -w 10

# save log to file
$ huey_consumer.py main.huey -k process -w 4 --logfile=../logs/huey.log

References

python