Async task with celery

A quick guide on how to use celery to accomplish async task in application.

Simple web stack with flask nginx gunicorn supervisor

A simple web stack using flask, nginx, gunicorn and supervisor in python with minimum configuration. This is useful if you want to QUICKLY setup a simple but production ready web service.

Redis db migration

A list of commands to do a quick migration for redis.

Flask based app consolidated logging

Flask has an “app.logger” which you can use for webapp related logging, and for other libraries or packages that flask app is using usually python standard logging is used. There are issues reported that logs from various sources in flask based app sometimes get missed out or get messy. See the references section at the bottom for details about the issues. A workaround is to set global root logging config and override that from flask.

Docker commands and files

Frequently used docker commands and examples of Dockerfile.

Huey as crontab alternative in python

Huey is a little task queue that supports:

  • multi-process, multi-thread or greenlet task execution models
  • schedule tasks to execute at a given time, or after a given delay
  • schedule recurring tasks, like a crontab
  • automatically retry tasks that fail
  • task result storage
  • consumer publishes event stream, allowing high-fidelity monitoring

Pathos solves PicklingError for multiprocssing

Python multiprocssing is useful in executing concurrent tasks with multiple processes. But it also requires the objects being executed support pickling, which is not always true for types like class instance methods, staticmethods and etc. Pathos has a multiprocessing implementation that uses dill on the backend which supports serializing and deserializing for almost all types.

Generate two dimensional arrays with serial numbers

Generate n*n two dimensional arrays with serial numbers in python.

SQL case statement

How to use CASE statement in sql select statement to deal with if-then-else logic.

Insert into mongodb with DBRef

How to insert a new object into mongodb using pymongo with reference to other objects using DBRef.