Discover BullMQ, a simple MessageQueue with NodeJS

Imagine a web application that needs to export a zip, then send it by email, so that a user can download data.

If your server waits for the zip to be compressed and then sent before responding to the user:

• the request becomes slow
• your server may block
• the request may fail due to a timeout

The solution is to delegate this work to an asynchronous task system.

Instead of creating the zip directly, the application adds a task to a queue:

(1) Export data (user@example.com) -- In progress...
(2) Export data (user2@test.com)   -- Waiting
(3) Export data (user3@fake.com)   -- Task added

The user will receive immediate confirmation that their request has been processed, and another process (sometimes another server, container, etc.) will take care of processing the request and sending the email.

Like this:

Read the current task
|
Retrieve the data
|
Create the zip archive
|
Send the email (user@example.com)
|
Complete the task

As long as there are tasks in progress, the process will handle them one after another, in order of arrival (First In - First Out or FIFO)

This way, our web server is not blocked by long or expensive tasks, and it can continue to easily handle the rest of its requests!

What is BullMQ?

BullMQ is a library for managing this type of task queue.

It relies on Redis, a very fast in-memory database used to store and coordinate tasks.

If you're not familiar with Redis, we've prepared a dedicated article for you!

The system works based on three elements:

• The queue: where tasks are added
• Workers: processes that execute the tasks
• Redis: the engine that stores jobs and their state

The flow generally looks like this:

1. The application adds a task to the queue
2. Redis stores this task
3. A worker retrieves the task
4. The worker executes the action (e.g., sending an email)

This model allows you to separate the main business logic from heavy processing.

Note that BullMQ is not a Message Broker, it's simply a task queue! We'll look at the difference with RabbitMQ, for example, a bit further on.

A simple example

Here's a minimal example with Node.js to add a task to a queue:

import { Queue } from "bullmq";

const emailQueue = new Queue("email-queue", {
  connection: {
    host: "localhost",
    port: 6379
  }
});

await emailQueue.add("send-welcome-email", {
  email: "user@example.com",
  name: "Alice"
});

This code adds a task called send-welcome-email to Redis.

And here's an example of a Worker that processes tasks:

import { Worker } from "bullmq";

const worker = new Worker(
  "email-queue",
  async job => {
    const { email, name } = job.data;

    console.log(`Envoi de l'email de bienvenue à ${name} (${email})`);

    // Ici on pourrait appeler un service SMTP
    // sendEmail(email, ...)
  },
  {
    connection: {
      host: "localhost",
      port: 6379
    }
  }
);

As soon as a task is added, the worker picks it up and executes the processing.

Typical use cases

Task queues like BullMQ are used for:

• sending emails
• generating reports or documents
• processing images or videos
• importing large amounts of data
• running scheduled tasks
• running heavy background processes

As soon as an operation might take time or fail temporarily, a queue becomes useful.

The strengths and weaknesses of BullMQ

BullMQ has several important advantages: it's performant, handles "retries" in case of failures, but above all has very good DX and only needs one infrastructure dependency (a Redis database).

But it remains a relatively simple solution nonetheless, with its limitations: limited to Node.js, Python, and PHP, it's also less suited to highly distributed architectures.

What about RabbitMQ then?

The main difference between BullMQ and RabbitMQ is their role:

• BullMQ is a task management library.
• RabbitMQ is a message broker, designed to enable multiple services to communicate with each other.

With RabbitMQ, each service can send messages to the queue, and each service is free to process a message as it sees fit. BullMQ simply allows you to send a task unilaterally, with a single source of truth (even if multiple Workers can listen for tasks).

RabbitMQ is also heavier to set up, requiring dedicated infrastructure and its AMQP-based protocol.

Additional note

It's important to remember that BullMQ is capable of keeping tasks in memory long-term only if persistence is enabled on your Redis database!