Redis as simple distribution lock for simple cases

Dec 25, 2023

I came across a system issue recently that some of our batch jobs written as ruby rake tasks ended up having duplicated invocations, which causes either duplicated records in database or duplicated alerts in slack. It is definitely an infra issue but until it is fixed from infra side, I need figure out a way to address it from the coding perspective. Trying to rewrite everything as idempotent as possible is not an option and neither is enhancing application level validations or database uniqueness constraints, because not all jobs can be idempotent and not all jobs write to databases, besides the issues are caused by race conditions not from two threads but from two processes. After some research and experiment, I decided to use Redis as a simple distribution lock solution, which is already available in our application and needs zero extra setup, and it is a single instance therefore no replication sync concern. Redis is single-threaded architecture so no matter how many concurrent clients try to write to it, it is guaranteed to be in sequential and atomic. I figure out I could make use of this nature to fix the issue faced.

Example of a task

namespace :main do
  task :hello do
    sleep 1 # simulate long-running task
    puts '***hello, world***'
  end
end

Run without lock

$ bundle exec rake main:hello & bundle exec rake main:hello &
[1] 38358
[2] 38359
***hello, world***
***hello, world***

[2]  + done       bundle exec rake main:hello
[1]  + done       bundle exec rake main:hello

Two processes run it simultaneously, and it runs twice.

Add redis distribution lock

• Define lock

namespace :lock do
  redis = Redis.new

  task :before do
    task_name = Rake.application.top_level_tasks[0]
    lock_obtained = redis.set task_name, '1', ex: 60, nx: true
    abort "Another process is already running #{task_name}" unless lock_obtained
  end

  task :after do
    task_name = Rake.application.top_level_tasks[0]
    redis.del task_name
  end
end

• Enhance the task with lock

Rake::Task['main:hello'].enhance(['lock:before']) do
  Rake::Task['lock:after'].execute
end

Run with lock

$ bundle exec rake main:hello & bundle exec rake main:hello &
[1] 38560
[2] 38561

Another process is already running main:hello
[1]  - exit 1     bundle exec rake main:hello

***hello, world***
[2]  + done       bundle exec rake main:hello

Two processes run it simultaneously, but it runs only once.

Summary

The point is to make use of the nx: true option of the set command, which means only set the key if it does not already exist. With redis single-threaded design, we can be sure that only one process will successfully set the key, denoting that only one process obtains the lock.

For referencee, Distributed Locks with Redis has more details on implementing more reliable and safer algorithem based locks such as Redlock for more complicated cases.

redis