IoT Telematics: CAN Data to the Cloud via an ESP32 Gateway (MQTT)

The problem

Fleet EVs report battery and fault data to the cloud, but the vehicle bus is CAN and the cloud speaks IP. A gateway must select and compress signals, survive patchy connectivity and not lose critical events, and its latency and data cost need to be measured, not assumed.

Objectives

• Receive and decode CAN 2.0 frames with hardware filters on the STM32.
• Design a compact UART protocol between the STM32 and an ESP32 gateway.
• Publish signals by MQTT with suitable topics, QoS and payload format.
• Buffer data in SPI flash during outages and forward it when the link returns.
• Measure end-to-end latency, loss and bytes per hour, and compare payload formats.

What you need

This project uses the kit's lithium cell. Respect polarity, never short the terminals, and don't leave the cell charging unattended.

Key concepts

Gateway

A node that connects two networks with different protocols, here CAN and MQTT over Wi-Fi, translating and filtering data between them.

MQTT QoS

QoS 0 sends at most once, QoS 1 at least once with acknowledgement (duplicates possible). The choice trades bandwidth against delivery guarantees.

bxCAN acceptance filters

Hardware filter banks on the STM32 that accept only the identifiers of interest, so the CPU is not interrupted by every frame on the bus.

Store-and-forward

When the uplink is down, data is saved locally with timestamps and sent later in order, so no event is lost.

Payload encoding

JSON is readable but large; a binary or CBOR-style payload is smaller. The difference matters on metered cellular links.

Step-by-step guide

1. 1

   Define the signal set and requirements

   Choose the CAN messages to emulate: BMS status (cell voltage, temperature, SOC, fault bits) at 100 ms, motor status (emulated speed, torque request) at 50 ms and a fault frame on change. Decide what goes to the cloud: periodic summaries every 5 s, faults immediately. Set requirements: maximum latency for faults, maximum data per hour, behaviour during outages. Write these as testable statements.

   Checkpoint: A signal list in DBC-style form and a requirements table that each later test refers to.

2. 2

   Build the two-node CAN bus

   Connect the kit's CAN screw terminal to the USB-CAN adapter (CAN_H to CAN_H, CAN_L to CAN_L, common ground) with 120 Ω at each end. In CubeMX enable CAN at 500 kbit/s (with a 36 MHz APB1 clock, prescaler 4 with BS1 = 15 and BS2 = 2 time quanta gives 500 kbit/s; verify with a bit-timing calculator). Set the DIP switch for the CAN block. Write a Python script with python-can to send the emulated traffic.

   Checkpoint: Frames from the PC are received by the STM32 and frames from the STM32 appear on the PC, with no error frames over 10 minutes.

3. 3

   Configure filters and reception

   Configure bxCAN filter banks with HAL_CAN_ConfigFilter in identifier-list or mask mode so only the chosen IDs reach FIFO0. Activate the RX FIFO 0 message-pending notification and read frames with HAL_CAN_GetRxMessage in the callback, pushing them into a ring buffer. Decode signals in the main loop, not in the interrupt. Track bus-off and error counters. Enable automatic bus-off recovery in the CAN init structure and log each recovery.

   Checkpoint: Frames with IDs outside the list are not received, and a counter shows no FIFO overruns at full traffic.

4. 4

   Design the STM32–ESP32 link

   Connect a spare STM32 USART to the ESP32 UART with jumper wires and a common ground (both are 3.3 V logic). Define binary frames: start byte, type, length, sequence number, RTC timestamp, payload and CRC-16. Add ACK frames from the ESP32 so the STM32 knows what was delivered to the broker. Run at 115200 baud or higher. Limit the frame length so a corrupted length byte cannot overflow the receive buffer.

   Checkpoint: A loopback test of 10,000 frames shows no CRC errors and every sequence number acknowledged.

5. 5

   Implement the ESP32 MQTT client

   On the ESP32, connect to Wi-Fi and to a Mosquitto broker on a laptop. Map frame types to topics such as fleet/vehicle01/bms and fleet/vehicle01/fault. Publish summaries at QoS 0 and faults at QoS 1, and send the ACK to the STM32 only when the broker acknowledges a QoS 1 publish. Use a last-will message so the cloud knows when the gateway goes offline.

   Checkpoint: Subscribing with mosquitto_sub shows live summaries and faults, and unplugging the ESP32 triggers the last-will message.

6. 6

   Add store-and-forward in SPI flash

   When ACKs stop arriving, write outgoing frames to a circular region in SPI flash with their timestamps. When the link returns, send backlog in order at a limited rate while still sending live faults first. Mark entries as delivered on ACK. Show link state and backlog size on the LCD. Size the buffer from the expected outage length and the data rate, and decide what to drop first if it fills: periodic summaries before faults. Record outage start and end times so the cloud side can mark them.

   Checkpoint: Turning off the Wi-Fi access point for five minutes and restoring it delivers every buffered record with original timestamps and no duplicates in the analysis.

7. 7

   Build the cloud-side logger

   Write a Python paho-mqtt subscriber that stores every message with broker receive time, decodes payloads and checks sequence numbers for gaps and duplicates. Plot signals over time and mark outage periods. Store messages in a small SQLite database or Parquet files so long runs can be queried easily. Compute delivery ratio and latency per topic automatically, and add a dashboard-style summary plot for each test run that shows signals, outages and backlog drain on the same time axis.

   Checkpoint: Gap and duplicate reports for a test run match what was injected.

8. 8

   Measure latency, loss and data volume

   Measure end-to-end latency from the CAN frame leaving the USB-CAN adapter to the MQTT message arriving, using the PC clock for both ends. Repeat for summaries and faults, with JSON payloads and with binary payloads, and at QoS 0 versus QoS 1. Count bytes on the wire per hour from broker statistics or packet capture. Synchronise the PC clock used for both ends, or measure both on the same PC.

   Checkpoint: A results table with median and 95th-percentile latency, loss and bytes per hour for each configuration.

9. 9

   Compare against a baseline

   Implement a naive baseline that forwards every received CAN frame as JSON without filtering or buffering. Compare data volume, latency and losses during outages with your gateway. Discuss which design choices gave the biggest benefit. Run both designs with the same scripted outage pattern, for example three outages of one, five and ten minutes, and the same CAN traffic file replayed by python-can. Report how many fault messages each design lost, how many bytes it used per hour and the latency percentiles.

   Checkpoint: A comparison table and plot showing the baseline against the optimised gateway under the same traffic and outage script.

10. 10

    Document and discuss security

    Write up the architecture, protocols, results and limitations. Discuss what production telematics would add: TLS on the MQTT connection, device certificates, and a separation between the vehicle bus and the gateway so the cloud cannot write arbitrary CAN frames. Include all scripts and firmware. Also note how credentials should be stored on the gateway and why downlink commands need authentication and rate limits. List the limitations of the testbed, such as a local broker instead of a real cloud service and Wi-Fi instead of cellular.

    Checkpoint: A report with architecture diagram, measured results and a security section.

Testing & validation

• No CAN frames lost at full emulated traffic (FIFO overrun counter stays at zero).
• Every fault message is delivered after a five-minute outage, verified by sequence numbers.
• Latency percentiles measured for each configuration.
• Data volume reduction against the naive baseline quantified.

Expected results

A working CAN-to-MQTT telematics chain with filtering, a reliable serial link, store-and-forward buffering and measured latency, loss and bandwidth compared with a naive forwarder.

Take it further

• Enable TLS on the ESP32 MQTT connection and measure its latency cost.
• Accept downlink commands such as changing the reporting rate, with authentication.
• Compress the backlog with delta encoding before forwarding.

Viva and review questions