MQTT in Elixir

Introduction

I recently read a good book titled "Building Weather Stations with Elixir and Nerves" , which describes how to introduce Elixir as a tool for building embedded applications.

By using Nerves, we can run Elixir code on a network-enabled device and interact with some control software.

The book mentioned above focuses primarily on Nerves, which uses the HTTP protocol for network interaction. While this is a reasonable choice in many cases, I would like to introduce another option that is widely used in production Internet of Things (IoT) setups: MQTT.

MQTT protocol

MQTT is a messaging protocol designed for Internet of Things (IoT) device communication. It is widely used in many fields such as banking, oil and gas, manufacturing, etc.

The MQTT protocol has many advantages, some of which are listed below:

• It is a lightweight binary protocol that typically runs on top of the TCP/IP protocol.
• It is designed for scenarios with unreliable networks and is ideal for outdoor installations.
• It follows the publish/subscribe pattern and simplifies the client side logic.

We'll demonstrate some of the advantages of MQTT in our setup.

MQTT Broker

An important feature of MQTT is that it simplifies client logic, which is critical for embedded devices. This is achieved through a publish/subscribe pattern: in MQTT, there is no concept of a "server". Instead, all participating entities are clients connected to the so-called broker . Client subscribes to topic , publishes message to them, broker does routing (and many other things).

A good broker for production, like EMQ X, usually provides not only MQTT routing but also many other interesting features, such as

• other types of connection methods, such as WebSockets;
• Different authentication and authorization modes;
• Stream data to the database;
• Custom routing rules based on message characteristics.

Sensor settings

For simplicity, our device will be represented by a normal Mix application: it can easily be converted to a Nerves application.

First, we create a Mix project:

mix new --sup weather_sensor
cd weather_sensor

In order to interact with the MQTT broker, we need an MQTT client. We use emqtt . Add this to mix.exs as a dependency:

defp deps do
  [
    {:emqtt, github: "emqx/emqtt", tag: "1.4.4", system_env: [{"BUILD_WITHOUT_QUIC", "1"}]}
  ]
end

We're going to put all the "sensor" code into the main module WeatherSensor, so we need to add it to the application manager lib/weather_sensor/application.ex:

defmodule WeatherSensor.Application do
  @moduledoc false

  use Application

  @impl true
  def start(_type, _args) do
    children = [
      WeatherSensor
    ]

    opts = [strategy: :one_for_one, name: WeatherSensor.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Now, let's implement the main module in lib/weather_sensor.ex:

defmodule WeatherSensor do
  @moduledoc false

  use GenServer

  def start_link([]) do
    GenServer.start_link(__MODULE__, [])
  end

  def init([]) do
    interval = Application.get_env(:weather_sensor, :interval)
    emqtt_opts = Application.get_env(:weather_sensor, :emqtt)
    report_topic = "reports/#{emqtt_opts[:clientid]}/temperature"
    {:ok, pid} = :emqtt.start_link(emqtt_opts)
    st = %{
      interval: interval,
      timer: nil,
      report_topic: report_topic,
      pid: pid
    }

    {:ok, set_timer(st), {:continue, :start_emqtt}}
  end

  def handle_continue(:start_emqtt, %{pid: pid} = st) do
    {:ok, _} = :emqtt.connect(pid)

    emqtt_opts = Application.get_env(:weather_sensor, :emqtt)
    clientid = emqtt_opts[:clientid]
    {:ok, _, _} = :emqtt.subscribe(pid, {"commands/#{clientid}/set_interval", 1})
    {:noreply, st}
  end

  def handle_info(:tick, %{report_topic: topic, pid: pid} = st) do
    report_temperature(pid, topic)
    {:noreply, set_timer(st)}
  end

  def handle_info({:publish, publish}, st) do
    handle_publish(parse_topic(publish), publish, st)
  end

  defp handle_publish(["commands", _, "set_interval"], %{payload: payload}, st) do
    new_st = %{st | interval: String.to_integer(payload)}
    {:noreply, set_timer(new_st)}
  end

  defp handle_publish(_, _, st) do
    {:noreply, st}
  end

  defp parse_topic(%{topic: topic}) do
    String.split(topic, "/", trim: true)
  end

  defp set_timer(st) do
    if st.timer do
      Process.cancel_timer(st.timer)
    end
    timer = Process.send_after(self(), :tick, st.interval)
    %{st | timer: timer}
  end

  defp report_temperature(pid, topic) do
    temperature = 10.0 + 2.0 * :rand.normal()
    message = {System.system_time(:millisecond), temperature}
    payload = :erlang.term_to_binary(message)
    :emqtt.publish(pid, topic, payload)
  end
end

and add some options in config/config.exs:

import Config

config :weather_sensor, :emqtt,
  host: '127.0.0.1',
  port: 1883,
  clientid: "weather_sensor",
  clean_start: false,
  name: :emqtt

config :weather_sensor, :interval, 1000

Let's summarize what's going on in the WeatherSensor:

• It implements the GenServer behavior.

• When started, it has the following actions:

  • Open an MQTT connection;
  • Subscribe to the commands/weather_sensor/set_interval topic to receive commands, and the received data will be sent to the process via :emqtt as {:publish, publish } messages.
  • Set a timer at predefined intervals.
• When the timer expires, it publishes a {Timestamp, Temperature} tuple to the reports/weather_sensor/temperature topic.
• It updates the timer interval when it receives a message from the commands/weather_sensor/set_interval topic.

Since our application is not a real Nerves application, it connects a sensor like a BMP280, so we generate temperature data.

Here we can already see an advantage over HTTP interaction: we can not only send data, but also receive some commands in real time.

We need a broker to run the node; we'll get to that later.

console settings

Since there is no "server" in MQTT, our console will also be an MQTT client. But it will subscribe to reports/weather_sensor/temperature topic and publish commands to commands/weather_sensor/set_interval.

For the console, we will set up the Phoenix LiveView application.

The creation process is as follows:

mix phx.new --version
Phoenix installer v1.6.2
mix phx.new weather_dashboard --no-ecto --no-gettext --no-dashboard --live
cd weather_dashboard

Add dependencies to mix.exs

  defp deps do
    [
      ...
      {:jason, "~> 1.2"},
      {:plug_cowboy, "~> 2.5"},

      {:emqtt, github: "emqx/emqtt", tag: "1.4.4", system_env: [{"BUILD_WITHOUT_QUIC", "1"}]},
      {:contex, github: "mindok/contex"} # We will need this for SVG charts
    ]
  end

Add some settings to config/dev.exs:

config :weather_dashboard, :emqtt,
  host: '127.0.0.1',
  port: 1883

config :weather_dashboard, :sensor_id, "weather_sensor"

# Period for chart
config :weather_dashboard, :timespan, 60

Now we generate a LiveView controller:

mix phx.gen.live Measurements Temperature temperatures  --no-schema --no-context

This generates a lot of files, but not all of them are necessary, what we need is a single page application with charts.

rm lib/weather_dashboard_web/live/temperature_live/form_component.*
rm lib/weather_dashboard_web/live/temperature_live/show.*
rm lib/weather_dashboard_web/live/live_helpers.ex

Also remove import WeatherDashboardWeb.LiveHelpers from lib/weather_dashboard_web.ex.

Update the template for our page (lib/weather_dashboard_web/live/temperature_live/index.html.heex):

<div>
  <%= if @plot do %>
    <%= @plot %>
  <% end %>
</div>

<div>
  <form phx-submit="set-interval">
    <label for="interval">Interval</label>
    <input type="text" name="interval" value={@interval}/>
    <input type="submit" value="Set interval"/>
  </form>
</div>

We have a chart and input controls that send commands to the "device" on this page.

Now update the main part of the LiveView controller (lib/weather_dashboard_web/live/temperature_live/index.ex):

defmodule WeatherDashboardWeb.TemperatureLive.Index do
  use WeatherDashboardWeb, :live_view

  require Logger

  @impl true
  def mount(_params, _session, socket) do
    reports = []
    emqtt_opts = Application.get_env(:weather_dashboard, :emqtt)
    {:ok, pid} = :emqtt.start_link(emqtt_opts)
    {:ok, _} = :emqtt.connect(pid)
    # Listen reports
    {:ok, _, _} = :emqtt.subscribe(pid, "reports/#")
    {:ok, assign(socket,
      reports: reports,
      pid: pid,
      plot: nil,
      interval: nil
    )}
  end

  @impl true
  def handle_params(_params, _url, socket) do
    {:noreply, socket}
  end

  @impl true
  def handle_event("set-interval", %{"interval" => interval_s}, socket) do
    case Integer.parse(interval_s) do
      {interval, ""} ->
        id = Application.get_env(:weather_dashboard, :sensor_id)
        # Send command to device
        topic = "commands/#{id}/set_interval"
        :ok = :emqtt.publish(
          socket.assigns[:pid],
          topic,
          interval_s,
          retain: true
        )
        {:noreply, assign(socket, interval: interval)}
      _ ->
        {:noreply, socket}
    end
  end

  def handle_event(name, data, socket) do
    Logger.info("handle_event: #{inspect([name, data])}")
    {:noreply, socket}
  end

  @impl true
  def handle_info({:publish, packet}, socket) do
    handle_publish(parse_topic(packet), packet, socket)
  end

  defp handle_publish(["reports", id, "temperature"], %{payload: payload}, socket) do
    if id == Application.get_env(:weather_dashboard, :sensor_id) do
      report = :erlang.binary_to_term(payload)
      {reports, plot} = update_reports(report, socket)
      {:noreply, assign(socket, reports: reports, plot: plot)}
    else
      {:noreply, socket}
    end
  end

  defp update_reports({ts, val}, socket) do
    new_report = {DateTime.from_unix!(ts, :millisecond), val}
    now = DateTime.utc_now()
    deadline = DateTime.add(DateTime.utc_now(), - 2 * Application.get_env(:weather_dashboard, :timespan), :second)
    reports =
      [new_report | socket.assigns[:reports]]
      |> Enum.filter(fn {dt, _} -> DateTime.compare(dt, deadline) == :gt end)
      |> Enum.sort()

    {reports, plot(reports, deadline, now)}
  end

  defp parse_topic(%{topic: topic}) do
    String.split(topic, "/", trim: true)
  end

  defp plot(reports, deadline, now) do
    x_scale =
      Contex.TimeScale.new()
      |> Contex.TimeScale.domain(deadline, now)
      |> Contex.TimeScale.interval_count(10)

    y_scale =
      Contex.ContinuousLinearScale.new()
      |> Contex.ContinuousLinearScale.domain(0, 30)

    options = [
      smoothed: false,
      custom_x_scale: x_scale,
      custom_y_scale: y_scale,
      custom_x_formatter: &x_formatter/1,
      axis_label_rotation: 45
    ]

    reports
    |> Enum.map(fn {dt, val} -> [dt, val] end)
    |> Contex.Dataset.new()
    |> Contex.Plot.new(Contex.LinePlot, 600, 250, options)
    |> Contex.Plot.to_svg()
  end

  defp x_formatter(datetime) do
    datetime
    |> Calendar.strftime("%H:%M:%S")
  end

end

Special instructions are as follows:

• We created a LiveView handler to serve our application's home page.
• Typically, Phoenix.PubSub is used to update the LiveView process status. However, we made a special setup: since the MQTT broker already provides a publish-subscribe model, we connect to it directly from the LiveView process.
• After receiving new temperature data, the server updates the temperature graph.
• The interval at which we send the update to the command subject after receiving a form update from the user.

Finally, set up routes in lib/weather_dashboard_web/router.ex so that our controller can handle the root page:

  scope "/", WeatherDashboardWeb do
    pipe_through :browser

    live "/", TemperatureLive.Index
  end

module integration

Now we're all set up and running.

We run an MQTT broker. Since we don't need any specific setup, the easiest way is to use docker to run the broker.

docker run -d --name emqx -p 1883:1883 emqx/emqx:4.3.10

Now run our "device":

cd weather_sensor
export BUILD_WITHOUT_QUIC=1
iex -S mix
Erlang/OTP 24 [erts-12.1.2] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit] [dtrace]

....

13:17:24.461 [debug] emqtt(weather_sensor): SEND Data: {:mqtt_packet, {:mqtt_packet_header, 8, false, 1, false}, {:mqtt_packet_subscribe, 2, %{}, [{"/commands/weather_sensor/set_interval", %{nl: 0, qos: 1, rap: 0, rh: 0}}]}, :undefined}

13:17:24.463 [debug] emqtt(weather_sensor): RECV Data: <<144, 3, 0, 2, 1>>

13:17:25.427 [debug] emqtt(weather_sensor): SEND Data: {:mqtt_packet, {:mqtt_packet_header, 3, false, 0, false}, {:mqtt_packet_publish, "/reports/weather_sensor/temperature", :undefined, :undefined}, <<131, 104, 2, 110, 6, 0, 179, 156, 178, 158, 125, 1, 70, 64, 38, 106, 91, 64, 234, 212, 185>>}

13:17:26.428 [debug] emqtt(weather_sensor): SEND Data: {:mqtt_packet, {:mqtt_packet_header, 3, false, 0, false}, {:mqtt_packet_publish, "/reports/weather_sensor/temperature", :undefined, :undefined}, <<131, 104, 2, 110, 6, 0, 156, 160, 178, 158, 125, 1, 70, 64, 39, 115, 221, 187, 144, 192, 31>>}
...

We saw that our sensor immediately started sending reports.

Now run our console:

cd weather_dashboard
export BUILD_WITHOUT_QUIC=1
iex -S mix phx.server
Erlang/OTP 24 [erts-12.1.2] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit] [dtrace]

[info] Running WeatherDashboardWeb.Endpoint with cowboy 2.9.0 at 127.0.0.1:4000 (http)
[info] Access WeatherDashboardWeb.Endpoint at http://localhost:4000
Interactive Elixir (1.12.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> [watch] build finished, watching for changes...

Let's navigate to http://localhost:4000 .

We see the corresponding LiveView process mount, connect to the proxy, and start receiving temperature data:

[info] GET /
[info] Sent 200 in 145ms
[info] CONNECTED TO Phoenix.LiveView.Socket in 129µs
  Transport: :websocket
  Serializer: Phoenix.Socket.V2.JSONSerializer
  Parameters: %{"_csrf_token" => "cwoROxAwKFo7NEcSdgMwFlgaZ1AlBxUa6FIRhAbjHA6XORIF-EUiIRqU", "_mounts" => "0", "_track_static" => %{"0" => "http://localhost:4000/assets/app.css", "1" => "http://localhost:4000/assets/app.js"}, "vsn" => "2.0.0"}
[debug] emqtt(emqtt-MacBook-Pro-iaveryanov-86405372ddbf17052130): SEND Data: {:mqtt_packet, {:mqtt_packet_header, 1, false, 0, false}, {:mqtt_packet_connect, "MQTT", 4, false, true, false, 0, false, 60, %{}, "emqtt-MacBook-Pro-iaveryanov-86405372ddbf17052130", :undefined, :undefined, :undefined, :undefined, :undefined}, :undefined}
[debug] emqtt(emqtt-MacBook-Pro-iaveryanov-86405372ddbf17052130): RECV Data: <<32, 2, 0, 0>>
[debug] emqtt(emqtt-MacBook-Pro-iaveryanov-86405372ddbf17052130): SEND Data: {:mqtt_packet, {:mqtt_packet_header, 8, false, 1, false}, {:mqtt_packet_subscribe, 2, %{}, [{"/reports/#", %{nl: 0, qos: 0, rap: 0, rh: 0}}]}, :undefined}
[debug] emqtt(emqtt-MacBook-Pro-iaveryanov-86405372ddbf17052130): RECV Data: <<144, 3, 0, 2, 0>>
[debug] emqtt(emqtt-MacBook-Pro-iaveryanov-86405372ddbf17052130): RECV Data: <<48, 58, 0, 35, 47, 114, 101, 112, 111, 114, 116, 115, 47, 119, 101, 97, 116,
  104, 101, 114, 95, 115, 101, 110, 115, 111, 114, 47, 116, 101, 109, 112, 101,
  114, 97, 116, 117, 114, 101, 131, 104, 2, 110, 6, 0, 180, 251, 188, 158, 125,
...

Also, the page starts updating immediately:

If we update the interval, we see that the device node immediately receives the command and starts updating more frequently:

Now we demonstrate one important thing: let's stop our "device" node, wait a moment, and restart it. We see that nodes continue to send data at an updated frequency.

How could this be? It's actually quite simple, the secret lies in the retain flag of the command message we send to the command topic.

:ok = :emqtt.publish(
  socket.assigns[:pid],
  topic,
  interval_s,
  retain: true
)

When we send a message with the retain flag to a topic, that message also becomes the "default" message and remains on the broker. Every subscriber to this topic will receive this message when subscribing.

This capability is important for embedded devices that may be offline frequently and do not have any easy-to-use local storage to maintain their state. Here's how to properly configure them when connecting.

in conclusion

This article describes the following:

• Demonstrates a popular way of interacting with embedded devices - the MQTT protocol;
• We introduced its usage in Elixir;
• We also demonstrated some of the advantages of MQTT, such as the publish-subscribe model and message retention.

Powerful features we might want to use even in simple setups are:

• Stream topic data into a database so we can display connection history without saving it "manually";
• Connect directly to the broker from the frontend using MQTT.js via WebSockets.

All code is available at https://github.com/savonarola/mqtt-article .