RedisSMQ - Yet another simple Redis message queue

RedisSMQ is a Node.js library for queuing messages (aka jobs) and processing them asynchronously with consumers. Backed by Redis, it allows scaling up your application with ease of use.

For more details about initial RedisSMQ design and the motivation behind it see https://medium.com/@weyoss/building-a-simple-message-queue-using-redis-server-and-node-js-964eda240a2a

Current MQ Architecture Overview

High-level overview of how RedisSMQ works:

• An application publishes messages using a producer.
• Consumers pull messages off queues and start processing.
• If an error occurs, messages are unacknowledged. Otherwise, once acknowledged, messages are moved to the

  acknowledged queue

  .
• Unacknowledged messages are re-queued with optional

  retryDelay

  . When

  retryThreshold

  is exceeded, messages are put in the

  deal-letter queue

  .

Features

• High-performance message processing
• Scalable: A queue can be consumed by many concurrent consumers, running on the same or different hosts.
• Persistent: No messages are lost in case of consumer failures.
• Atomic: A message can be delivered only to one consumer at a time.
• Message expiration: A message will expire and not be consumed if it has been in the queue for longer than the TTL (time-to-live).
• Message consume timeout: The amount of time for a consumer to consume a message. If the timeout exceeded, message processing is cancelled and the message is re-queued again.
• Delaying and scheduling message delivery: Messages can be configured to be delayed, delivered for N times with an optional period between deliveries, and to be scheduled using CRON expressions.
• Reliable Priority Queues: Supports priority messaging.
• HTTP API: an HTTP interface is provided to interact with the MQ.
• Web UI: The MQ state (input/processing/acks/unacks messages rates, online consumers, queues, etc.) is provided and updated in real-time. The Web UI allows monitoring and managing the MQ.
• JSON Logging: Supports JSON log format for troubleshooting and analytic purposes.
• Highly optimized: No promises, no async/await, small memory footprint, no memory leaks. See callbacks vs promises vs async/await benchmarks.
• Configurable: Many options and features can be configured.
• Rigorously tested: With 79+ tests and code coverage no less than 80%.
• Supports both redis & ioredis: RedisSMQ can be configured to use either

  redis

  or

  ioredis

  to connect to Redis server.
  

Table of content

1. What's new?
2. Installation
3. Configuration
4. Usage
   1. Basic
      1. Message Class
      2. Producer Class
      3. Consumer Class
   2. Advanced Topics
      1. Message Scheduler
      2. Priority Queues
      3. Message Manager
      4. Queue Manager
      5. HTTP API
      6. Web UI
      7. Logs
5. Performance
6. Contributing
7. License

What's new?

2021.11.02

• v4 is out with significant performance improvements and new features including the ability to fetch/delete/requeue messages from different queues using the

  MessageManager

  /

  QueueManager

  or with the help of the HTTP API. If you are upgrading your installation, take a look at the migration guide before proceeding.

• The next release will allow the message management features to be used from the

  Web UI

  .

2021.10.07

• With the release of v3.3, reliable, persistent priority queues are now supported.

See CHANGELOG for more details.

Installation

npm install redis-smq --save

Considerations:

• RedisSMQ is targeted to be used in production environments. Therefore, only active LTS and maintenance LTS Node.js releases (v12, v14, and v16) are supported. The latest stable Node.js version is recommended.
• Minimal Redis server version is 2.6.12.

Configuration

See Configuration for more details.

Usage

Basics

RedisSMQ provides 3 classes: Message, Producer and Consumer in order to work with the message queue.

Message Class

Message class is the main component responsible for creating and handling messages. It encapsulates and provides all the required methods needed to construct and deal with messages.

const { Message } = require('redis-smq');
const message = new Message();
message
    .setBody({hello: 'world'})
    .setTTL(3600000)
    .setScheduledDelay(10000) // in millis
    .setScheduledRepeat(6)
    .setScheduledPeriod(60000)
    .setScheduledCron('* 30 * * * *');

let messageTTL = message.getTTL();

See Message Reference for more details.

Producer Class

Producer class is in turn responsible for producing messages.

Each producer instance has an associated message queue and provides

produceMessage()

// filename: ./examples/javascript/ns1-test-queue-producer.js

'use strict';
const { Message, Producer } = require('redis-smq');
const message = new Message();
message
    .setBody({hello: 'world'})
    .setTTL(3600000);
const producer = new Producer('test_queue');
producer.produceMessage(message, (err) => {
   if (err) console.log(err);
   else console.log('Successfully produced')
});

See Producer Reference for more details.

Consumer Class

The Consumer class is the base class for all consumers. All consumers extends this base class and implements

consume()

Consumer classes are saved per files. Each consumer file represents a consumer class.

A consumer class may look like:

// filename: ./examples/javascript/ns1-test-queue-consumer.js
'use strict';

const { Consumer } = require('redis-smq');
class TestQueueConsumer extends Consumer {
    consume(message, cb) {
        console.log('Got a message to consume:', message);
        cb();
    }
}
const consumer = new TestQueueConsumer('test_queue');
consumer.run();

To start consuming messages, a consumer needs first to be launched from CLI to connect to the Redis server and wait for messages:

$ node ./examples/javascript/test-queue-consumer.js

Once a message is received and processed the consumer should acknowledge the message by invoking the callback function without arguments.

The message acknowledgment informs the MQ that the message has been successfully consumed.

If an error occurs, the message should be unacknowledged and the error should be reported to the MQ by calling the callback function. Failed messages are re-queued and delivered again unless message retry threshold is exceeded. Then the messages are moved to dead-letter queue (DLQ). Each message queue has a system generated corresponding queue called dead-letter queue where all failed messages are moved to.

See Consumer Reference for more details.

Advanced Topics

• Scheduler

• Priority Queues

• Message Manager

• Queue Manager

• HTTP API

• Web UI

• Logs

Performance

See Performance for more details.

Contributing

So you are interested in contributing to this project? Please see CONTRIBUTING.md.

License

MIT