Software

How to use Message Queuing Telemetry Transport

by Professor Petabyte

Introduction

MQTT stands for Message Queuing Telemetry Transport. It is a lightweight messaging protocol designed for constrained devices and unreliable networks, making it ideal for Internet of Things (IoT) applications.

Key:

This page includes lots of code samples. For clarity, they have colour coded backgroungs like this:-

Commands to be run on a Linux command line.

Python code.

SQL code/queries

HTML script

Anything outside of the above categories

About MQTT

Mouseover to Zoom

• Lightweight:
  MQTT is designed to be small and efficient, requiring minimal resources from devices, which is crucial for resource-constrained IoT devices.


• Messaging Protocol:
  It facilitates communication between devices by using a publish-subscribe model.


• Publish-Subscribe Model:
  In this model, devices (publishers) send messages to a central server (broker) without needing to know which devices are subscribed to receive those messages. Subscribers, in turn, receive messages relevant to their subscriptions from the broker.


• Constrained Devices:
  MQTT is particularly well-suited for devices with limited processing power, memory, and bandwidth, such as sensors, actuators, and embedded systems.


• Unreliable Networks:
  It works well in environments with intermittent or unstable network connections, making it a reliable choice for remote or mobile IoT deployments.


• IoT Applications:
  MQTT is widely used in various IoT applications, including smart homes, industrial automation, and connected vehicles.

Essentially, MQTT provides a simple and efficient way for diverse devices to exchange information, even in challenging network conditions, which is essential for the growth and functionality of the IoT ecosystem.

Mouseover to Zoom

How MQTT Works

1. Broker (aka Central Server)
   
   • Acts like a mail router.
   • Receives messages from Publishers and passes thjem to all Subscribers to the specified Topic.


2. Publisher
   
   • Sends messages about specified topics.
   • Does not need to know who is subscribed to the specified Topic.


3. Subscriber
   
   • Listens for messages on specific topic(s).
   • Does not need to know who is sending the messages (or who else is subscribed.

Topics

• Topics are hierarchical strings, like file paths
  

  home/bedroom/fan

• Clients can subscribe to the Publisher using wildcards. e.g.

  • home/+/fan

    matches all fans within home, whilst

  • home/bedroom/#

    matches everything in bedroom

Quality of Service (QoS)

MQTT Quality of Service (QoS) controls message delivery guarantees between client and broker. It's important when reliability matters — for example, in sensor networks or industrial systems where data loss or duplication can be a problem.

There are three levels of service:

1. QoS 0 - Send at Most Once (Unreliable)
   No guarantee of delivery. Each message is sent once only, no retries. No acknowledgement from broker. Fastest and lowest overhead. This should be used when an occasional message loss is acceptable, e.g. Temperature updates every second.
   


2. QoS 1 - At Least Once (Reliable but May Duplicate)
   Broker must acknowledge receipt. If no ack, publisher resends until it gets one. Receiver might get duplicate messages. This should be used when it is critical that the message is received, even if it's received more than once. e.g. Alarm triggers, control commands, etc.
   


3. QoS 2 - Exactly Once (Most Reliable)
   Two-phase handshake ensures no duplicates. Publisher and broker do extra work to ensure exactly one delivery. Most reliable, but slower and heavier. This should be used when Duplicate messages could cause serious problems, e.g. Banking transactions, inventory control.
   

How QoS is specified depends on the client library being used. Here are two common examples:

Python (Paho MQT)

import paho.mqtt.client as mqtt

client = mqtt.Client()
client.connect("broker.hivemq.com", 1883)

# Publish with QoS 1
client.publish("home/light", "on", qos=1)

# Subscribe with QoS 2
client.subscribe("home/light", qos=2)

MicroPython (umqtt.simple / robust)

For umqtt.simple (QoS 0 only) — you need umqtt.robust or another library for higher QoS

client.subscribe(b"home/light", qos=1)  # if supported
client.publish(b"home/light", "on", qos=1)

N.B. Not all libraries or brokers support QoS 2, especially in constrained devices like microcontrollers.

A Demonstration

There are many ways in which MQTT can be demonstrated / used / tested. This is a step-by-step example of how it can be demonstrated on a standard Ubuntu PC.

1. Update your package list

sudo apt update

2. Install Mosquitto and the client tools

sudo apt install -y mosquitto mosquitto-clients

3. Enable Mosquitto to start on boot

sudo systemctl enable mosquitto

4. Start the Mosquitto service

sudo systemctl start mosquitto

5. Test the installation
   Open two terminal windows; One to subscribe to a topic and the other to publish something to that topic.
   
   Terminal 1

   mosquitto_sub -h localhost -t test/topic

   
   Terminal 2

   mosquitto_pub -h localhost -t test/topic -m "What hath God wrought"

   
   The message specified in Terminal 2 should appear in Terminal 1.

Optional Extras

If you want to access the MQTT broker remotely, Open MQTT Port (1883) in Firewall

sudo ufw allow 1883

A username and password can be set in the /etc/mosquitto/mosquitto.conf file and generate credentials with:

sudo mosquitto_passwd -c /etc/mosquitto/passwd myuser

Walk through setting up a secure Mosquitto MQTT broker on Ubuntu

This is a walk through setting up a secure Mosquitto MQTT broker on Ubuntu with username/password authentication and TLS encryption.

Step-by-Step: Secure MQTT with Password & TLS

1. Install Mosquitto and clients (if not done) sudo apt update

   sudo apt install -y mosquitto mosquitto-clients



2. Create a password file

sudo mosquitto_passwd -c /etc/mosquitto/passwd myuser

• Replace myuser with your desired username.
• You'll be prompted for a password.
• To add more users later, remove the -c flag.


3. Configure Mosquitto to use authentication
Edit the configuration file:

sudo nano /etc/mosquitto/conf.d/default.conf

Paste the following minimal secure config:

allow_anonymous false
password_file /etc/mosquitto/passwd

listener 1883

You can add listener 8883 and TLS settings later for encrypted comms.



4. Restart Mosquitto

sudo systemctl restart mosquitto



5. Test authentication
Try subscribing with credentials:

mosquitto_sub -h localhost -t test/topic -u myuser -P yourpassword

And in another terminal, publish:

mosquitto_pub -h localhost -t test/topic -u myuser -P yourpassword -m "What hath God wraught"

If you see the message, authentication works.

Enable TLS Encryption (Optional but recommended)

6. Create self-signed certificates (or use Let's Encrypt)

sudo mkdir -p /etc/mosquitto/certs
cd /etc/mosquitto/certs

sudo openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -keyout mosquitto.key -out mosquitto.crt \
  -subj "/CN=yourdomain.com"



7. Update config to use TLS
Add to /etc/mosquitto/conf.d/default.conf

listener 8883
cafile /etc/mosquitto/certs/mosquitto.crt
certfile /etc/mosquitto/certs/mosquitto.crt
keyfile /etc/mosquitto/certs/mosquitto.key
require_certificate false



8. Restart Mosquitto

sudo systemctl restart mosquitto



9. Test with TLS connection

mosquitto_sub --cafile /etc/mosquitto/certs/mosquitto.crt -h localhost -p 8883 -t test/topic -u myuser -P yourpassword

Python example using paho-mqtt

Here's a that securely subscribes and publishes to your Mosquitto MQTT broker with username/password and optional TLS.

1. Install the Paho MQTT library
IF you haven't installed it yet:

pip install paho-mqtt



2. Secure MQTT Publisher (Python)

import paho.mqtt.client as mqtt

# Broker config
broker = "localhost"
port = 8883  # TLS port
topic = "secure/topic"
username = "myuser"
password = "yourpassword"

# TLS config
cafile = "/etc/mosquitto/certs/mosquitto.crt"  # path to your broker's cert

# Create client instance
client = mqtt.Client()

# Set credentials
client.username_pw_set(username, password)

# Enable TLS
client.tls_set(cafile=cafile)

# Connect and publish
client.connect(broker, port)
client.publish(topic, "Hello from secure Python client!")
client.disconnect()



3. Secure MQTT Subscriber (Python)

import paho.mqtt.client as mqtt

broker = "localhost"
port = 8883
topic = "secure/topic"
username = "myuser"
password = "yourpassword"
cafile = "/etc/mosquitto/certs/mosquitto.crt"

# Callback when a message is received
def on_message(client, userdata, msg):
    print(f"Received: {msg.topic} -> {msg.payload.decode()}")

client = mqtt.Client()
client.username_pw_set(username, password)
client.tls_set(cafile=cafile)

client.on_message = on_message

client.connect(broker, port)
client.subscribe(topic)

print("Listening for secure messages...")
client.loop_forever()

Notes:

• Make sure the cafile path matches the certificate used by your Mosquitto broker.
• These examples assume the broker is on the same machine (localhost). Use your server's IP or domain if remote.

Secure Python MQTT Subscriber with Mosquitto Broker writing to a MariaDB Database

Below is a secure Python MQTT subscriber that connects to your Mosquitto broker, receives messages, and inserts them into a MariaDB database.

Prerequisites

1. Install required Python packages

pip install paho-mqtt mysql-connector-python



2. MariaDB Setup
Create a database and table to store MQTT messages:

CREATE DATABASE mqtt_data;

USE mqtt_data;

CREATE TABLE messages (
    id INT AUTO_INCREMENT PRIMARY KEY,
    topic VARCHAR(255),
    payload TEXT,
    received_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Python MQTT Subscriber + MariaDB Insert

import paho.mqtt.client as mqtt
import mysql.connector
from datetime import datetime

# MQTT configuration
broker = "localhost"
port = 8883
topic = "secure/topic"
username = "myuser"
password = "yourpassword"
cafile = "/etc/mosquitto/certs/mosquitto.crt"

# MariaDB configuration
db_config = {
    "host": "localhost",
    "user": "mqtt_user",          # Create this user with necessary rights
    "password": "mqtt_password",  # Replace with your real password
    "database": "mqtt_data"
}

# Callback: when message is received
def on_message(client, userdata, msg):
    payload = msg.payload.decode()
    print(f"[{datetime.now()}] {msg.topic} => {payload}")
    
    # Save to MariaDB
    try:
        db = mysql.connector.connect(**db_config)
        cursor = db.cursor()
        sql = "INSERT INTO messages (topic, payload) VALUES (%s, %s)"
        cursor.execute(sql, (msg.topic, payload))
        db.commit()
        cursor.close()
        db.close()
    except mysql.connector.Error as err:
        print(f"Database error: {err}")

# Setup MQTT client
client = mqtt.Client()
client.username_pw_set(username, password)
client.tls_set(cafile=cafile)

client.on_message = on_message

client.connect(broker, port)
client.subscribe(topic)

print("MQTT subscriber connected and logging to MariaDB...")
client.loop_forever()

Database Permissions

CREATE USER 'mqtt_user'@'localhost' IDENTIFIED BY 'mqtt_password';
GRANT INSERT, SELECT ON mqtt_data.* TO 'mqtt_user'@'localhost';
FLUSH PRIVILEGES;

Final Test

1. Run the Python script.
2. Publish to MQT

mosquitto_pub -h localhost -p 8883 --cafile /etc/mosquitto/certs/mosquitto.crt \
-u myuser -P yourpassword -t secure/topic -m "Hello MariaDB!"

3. Check your MariaDB:

SELECT * FROM mqtt_data.messages;

This can be extended to also retrieve and display data in a web page. PHP or a framework such as FLASK can be used for this. The following is how to do it with FLASK, the Python framework for Web apps.

FLASK Prerequisites

Install Flask and MySQL connector

pip install flask mysql-connector-python

Project Structure (example)

mqtt_viewer/
├── app.py
└── templates/
    └── index.html

app.py - Flask Web App

from flask import Flask, render_template
import mysql.connector

app = Flask(__name__)

# MariaDB connection config
db_config = {
    "host": "localhost",
    "user": "mqtt_user",
    "password": "mqtt_password",
    "database": "mqtt_data"
}

@app.route("/")
def index():
    try:
        db = mysql.connector.connect(**db_config)
        cursor = db.cursor(dictionary=True)
        cursor.execute("SELECT * FROM messages ORDER BY received_at DESC LIMIT 100")
        messages = cursor.fetchall()
        cursor.close()
        db.close()
    except mysql.connector.Error as err:
        return f"<h1>Database Error</h1><p>{err}</p>"

    return render_template("index.html", messages=messages)

if __name__ == "__main__":
    app.run(debug=True)

templates/index.html - HTML Page

<!doctype html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>MQTT Message Viewer</title>
    <style>
        body { font-family: Arial, sans-serif; margin: 40px; }
        table { border-collapse: collapse; width: 100%; }
        th, td { border: 1px solid #ccc; padding: 8px; }
        th { background: #f4f4f4; }
    </style>
</head>
<body>
    <h1>MQTT Messages</h1>
    <table>
        <tr>
            <th>ID</th>
            <th>Topic</th>
            <th>Payload</th>
            <th>Received At</th>
        </tr>
        {% for msg in messages %}
        <tr>
            <td>{{ msg.id }}</td>
            <td>{{ msg.topic }}</td>
            <td>{{ msg.payload }}</td>
            <td>{{ msg.received_at }}</td>
        </tr>
        {% endfor %}
    </table>
</body>
</html>

Run the App

In the mqtt_viewer/ folder:

python app.py

Open a brower and go to

http://localhost:5000

Conclusion

By combining all these elements, MTQQ, Bash scripts, MySQL, Python and FLASK, it is possible to very quickly and easily create a way to capture data with either a single or a network of Raspberry Pi Picos, wirelessly transmit the data in real time to a Raspberry Pi, store it in an SQL database, and do all kinds of other data processing & analysis. The sensors could be logging all kinds of data:

• Environmental - heat, temperature, humidity, light intensity,etc
• System activity - pumps, lamps, motors etc.
• Active Sensors - microswitches on doors, windows, etc

MTQQ can also be used to send messages between networked computers, e.g. a simple messaging service between co-workers, e.g.
"Are you ready to go to lunch?" -- reply: "Almost, give me ten minutes."