Simple job queues for Python
RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It should be integrated in your web stack easily.
RQ requires Redis >= 3.0.0.
Full documentation can be found here.
If you find RQ useful, please consider supporting this project via Tidelift.
First, run a Redis server, of course:
To put jobs on queues, you don't have to do anything special, just define your typically lengthy or blocking function:
import requests def count\_words\_at\_url(url): """Just an example function that's called async.""" resp = requests.get(url) return len(resp.text.split())
You do use the excellent requests package, don't you?
Then, create an RQ queue:
from redis import Redis from rq import Queue q = Queue(connection=Redis())
And enqueue the function call:
from my\_module import count\_words\_at\_url job = q.enqueue(count\_words\_at\_url, 'http://nvie.com')
For a more complete example, refer to the docs. But this is the essence.
To start executing enqueued function calls in the background, start a worker from your project's directory:
$ rq worker \*\*\* Listening for work on default Got count\_words\_at\_url('http://nvie.com') from default Job result = 818 \*\*\* Listening for work on default
That's about it.
Simply use the following command to install the latest released version:
pip install rq
If you want the cutting edge version (that may well be broken), use this:
pip install -e git+https://github.com/nvie/[email protected]#egg=rq
Check out these below repos which might be useful in your rq based project.
This project has been inspired by the good parts of Celery, Resqueand this snippet, and has been created as a lightweight alternative to the heaviness of Celery or other AMQP-based queueing implementations.