Scale IoT backends horizontally with MQTT Shared Subscriptions

Diario del capitán, fecha estelar d37.y41/AB

mqtt iot
Fundador & CTO
Scale IoT backends horizontally with MQTT Shared Subscriptions

In IoT architectures, one core requirement is the ability to process all the messages sent by the connected devices in a scalable manner.

Choosing the right broker is a crucial part of the decision that will help achieve this goal, as discussed in my last post. However, it's equally important to select the appropriate strategy for consuming all the data processed by the message broker.

Whenever possible, I recommend using message broker rules. Depending on the message broker, they can be more or less customizable. From the power of the EMQX Rule Engine to the flexibility of AWS IoT.

One common use for message broker rules is to create different layers of message brokers, the will publish to other message brokers depending on the kind of message received from the devices. This allows the platform to scale the processing of each message type independently. For example, this is what we used when we collaborated on the Spin project.

When message broker rules do not fit your needs, you can always rely on messaging protocol strategies to scale the message processing.

MQTT Shared Subscriptions

MQTT 5.0 introduced shared subscriptions. All IoT message brokers that are fully compliant with MQTT 5.0, such as Mosquitto or EMQX, support shared subscriptions.

Shared subscriptions enable multiple MQTT clients to subscribe to the same topic. However, instead of each client receiving every message sent by a device, the message broker will load balance and distribute the messages among all subscribers using various configurable strategies.

Using shared subscriptions is quite straightforward. Here is the code of one of our applications, built with Node.js and TypeScript, using the mqtt.js client.

  const client = connect(brokerUrl, {

  const topic = `$share/consumer-servers/${topicName}`;

  client.on('connect', () => {
    client.subscribe(topic, (error: unknown) => {
        ? logError('Could not subscribe to topics', error)
        : logSuccess('Subscribed to topics');

All subscribers that start with $share/group-name/ will form a group that will be load balanced by the message broker. This allows us to scale all the nodes of the consumer-servers group as much as we need, depending on the devices connected and the time required to process every message.

If we need to scale different groups independently, we can do it by defining more groups, such as $share/expensive-consumer-servers/

Using MQTT Shared subscriptions instead of message broker rules has allowed us to implement complex behaviours in a multitenant application in a simple way, while reusing part of the data layer backend code developed in the frontend and the API of our application.

If you have any doubts on what strategy could be the most appropiate for your IoT application, write me a message on X and I can share my experience with you!

Compartir este post

Artículos relacionados

Robot staring at the camera - possibly your next CTO

Why does your startup not need a CTO (just yet)

Going for a CTO way too early in the life of your startup may be detrimental to your success. Learn why.

Leer el artículo
How I use Docker for Rails development: running services

How I use Docker for Rails development: running services

Project setup can be a very cumbersome process for developers. In this blog post, our developer Dani explains how he uses Docker to develop in Rails

Leer el artículo
MQTT Broker for IOT

Comparison of MQTT Brokers for IoT

In our recent IoT project for Ecler, we designed their cloud platform and supporting systems for connected amplifiers and sound systems. During the development, we tested three message broker solutions: Mosquitto, NanoMQ, and EMQX. Here are the pros and cons we found for each one of them.

Leer el artículo