The problem
Data loggers and telematics units on low-cost controllers have little memory and bandwidth. Vehicle signals are smooth and correlated, so they compress well, but general-purpose compressors are often too heavy for a 20 KB RAM MCU without an FPU. Choosing a codec needs measured trade-offs.
Objectives
- Record a representative dataset of kit signals (IMU, pressure, cell voltage, temperature, throttle, emulated speed).
- Implement delta encoding, zigzag-varint, Rice/Golomb coding and a block-adaptive variant in C.
- Port an embedded LZ-style compressor as a general-purpose baseline.
- Measure compression ratio, cycles per sample, RAM and code size on the STM32.
- Verify losslessness by decoding on the PC and comparing bit for bit.
What you need
From the EV Embedded Development Kit
- SPI flash memory
- Serial UART interface
- MPU6050 accelerometer and gyroscope
- BMP280 pressure and temperature sensor
- Analog potentiometer input
- 1S battery with connector and T-sense input
- STM32F103C8T6 main controller
Software
- STM32CubeIDE with HAL drivers
- Embedded C
- An embedded LZ-style compression library such as heatshrink (for the baseline)
- Python 3 with NumPy, pandas and matplotlib for decoding and analysis
Extra parts
None. Everything this project needs is on the kit.
This project uses the kit's lithium cell. Respect polarity, never short the terminals, and don't leave the cell charging unattended.
Key concepts
- Delta encoding
- Storing the difference between consecutive samples instead of the sample. Smooth signals give small differences that need fewer bits.
- Zigzag and varint
- Zigzag maps signed differences to unsigned numbers (0, -1, 1, -2 becomes 0, 1, 2, 3) and varint stores small numbers in fewer bytes.
- Rice coding
- A simple entropy code for small non-negative integers using a quotient in unary and a remainder in k bits. Choosing k per block adapts it to the signal.
- Cycle counting
- The Cortex-M3 DWT cycle counter (DWT->CYCCNT) gives exact CPU cycles for a piece of code, which is how codec cost is measured.
- Entropy bound
- The empirical entropy of the residuals gives a lower bound on bits per sample for a memoryless code, against which each codec is compared.
Step-by-step guide
- 1
Collect a representative dataset
Log raw 16-bit samples of the six IMU axes at 100 Hz, BMP280 pressure and temperature at 10 Hz, cell voltage and temperature at 1 Hz, and throttle and emulated speed at 10 Hz, over a range of scenarios: static, handled, throttle sweeps, and tapping. Stream raw data over UART to the PC and save it, so every codec is tested on identical input. Note that speed is emulated from the potentiometer.
Checkpoint: A dataset of at least an hour across scenarios, documented with sample rates, units and scenario labels.
- 2
Analyse the signals on the PC
In Python compute, per channel, the distribution of first and second differences, their empirical entropy in bits per sample and their autocorrelation. This tells you which predictor (previous value, linear extrapolation) suits which channel and sets the theoretical target for each codec. Also plot residual histograms per scenario, because a codec tuned for a static board may do badly during taps or throttle sweeps. Note channels where noise dominates, such as the least significant bits of the IMU, since these set a floor on lossless compression.
Checkpoint: A table of raw bits, entropy of first differences and entropy of second differences per channel.
- 3
Prototype codecs in Python
Implement delta plus zigzag-varint, delta plus Rice coding with a fixed k, and block-adaptive Rice (choose the best k for each 64-sample block and store it in a small header). Decode each and compare with the original to prove losslessness. Compute compression ratio per channel and scenario. Include the block header size in the compressed size so ratios are honest, and test several block lengths (32, 64, 128 samples) to see the trade-off between adaptivity and header overhead.
Checkpoint: Every codec decodes the full dataset bit-exactly in Python, with ratios tabulated.
- 4
Set up the embedded benchmark
In CubeMX configure SPI for the flash, UART, and the sensors used. Enable the DWT cycle counter (set CoreDebug->DEMCR TRCENA, then DWT->CTRL CYCCNTENA). Build with a fixed optimisation level (for example -O2) and record it. Create a test harness that feeds recorded blocks to each codec from a buffer received over UART, so on-board results use the same data as the PC.
Checkpoint: The harness reports cycles for an empty function call, confirming the counter works and giving its overhead.
- 5
Implement the codecs in C
Write a bit-writer that packs bits into a byte buffer, then the delta, zigzag, varint and Rice encoders using only integer operations. Keep the state per channel small. Measure code size from the map file and static RAM. Make the encoder work block by block so the output can be written to flash in 256-byte pages. Write a unit test that encodes a known vector and compares the output bytes with the Python version.
Checkpoint: Encoded output from the STM32, sent to the PC, decodes bit-exactly with the Python decoder.
- 6
Port the general-purpose baseline
Port an embedded LZ-style library such as heatshrink with a small window (for example 2^8 bytes) to fit RAM. Compress the same blocks, decode on the PC and measure ratio, cycles and RAM. Also compress delta-encoded data with it to see whether combining helps. Record the library version and configuration used. Check that its static and stack memory fit alongside the logger's buffers and drivers within 20 KB, and note whether its decoder would be needed on the device or only on the PC.
Checkpoint: Baseline results recorded for raw and delta-encoded input, with RAM usage noted.
- 7
Benchmark cycles and ratio
For each codec, channel and scenario record compression ratio, cycles per sample (mean and worst case), stack use and code size. Express CPU cost as a percentage of a 72 MHz core at the logging rate. Plot ratio against cycles to show the trade-off frontier. Run each measurement several times and report the spread. Disable interrupts other than those the logger needs during the measurement so the counts are not inflated by unrelated code, and state this in the method.
Checkpoint: A trade-off plot and a table with all metrics for every codec.
- 8
Integrate with the flash logger
Pick the best codec per channel and integrate it into a live logger that compresses blocks and writes them to SPI flash with a block header (channel, start time, sample count, codec parameters, CRC). Make sure a corrupted block affects only itself. Measure how much longer logging lasts compared with raw storage. Check that compression of one block finishes before the next block is full, using the worst-case cycle count, so samples are never dropped.
Checkpoint: A live log decodes on the PC with every block's CRC passing and the extended logging time calculated.
- 9
Stress-test robustness
Test worst cases: noisy signals, sudden steps (taps), sensor saturation and constant values. Check that the encoder never overflows its output buffer and falls back to raw storage for a block if compression would expand it. Generate synthetic worst cases on the PC as well, such as white noise and alternating full-scale values, and feed them through the UART harness. Record the compressed size and cycles for each, and confirm that decode still works and the CRC still passes.
Checkpoint: No buffer overflows across stress inputs, and expansion is capped by the raw fallback.
- 10
Write up and discuss
Present the dataset, entropy analysis, codecs, benchmark method and results. Discuss which codec you recommend for which signal type and the cost of each in cycles and RAM. Archive code, data and notebooks. Explain the method well enough that someone else could rerun the benchmark: compiler version, optimisation flags, clock speed, block sizes and the exact dataset. Discuss threats to validity, such as how representative the desk-generated signals are of real vehicle data.
Checkpoint: A report with reproducible results and a clear recommendation backed by the trade-off plot.
Testing & validation
- Bit-exact round trip for every codec on the full dataset.
- Cycle counts measured with DWT, with harness overhead subtracted.
- Raw fallback limits worst-case expansion to the block header size.
- Live logger blocks pass CRC and decode correctly.
Expected results
A measured comparison of lightweight lossless codecs on real vehicle-style signals, with a recommended per-channel scheme that extends flash logging time at a known CPU cost.
Research angle for your M.Tech thesis
Research questions: how close can very light integer codecs come to the entropy bound for vehicle signals, and where on the ratio–cycles frontier do they sit compared with a general-purpose LZ compressor? Metrics: compression ratio, cycles per sample, RAM, code size and robustness. Baseline: raw storage and heatshrink. A publishable contribution would release the dataset and benchmarks, and propose an adaptive per-block scheme justified by the measured frontier.
Take it further
- Add a linear or adaptive predictor before the residual coder.
- Use the same codec to reduce telematics payload size over Bluetooth or MQTT.
- Compare with near-lossless coding where a bounded error is allowed.
Viva and review questions
Why does delta encoding help?
Consecutive samples of smooth signals are close, so their differences are small numbers that need fewer bits to store.
Why zigzag before varint?
Differences can be negative; zigzag maps them to small unsigned numbers so varint can store them compactly.
How did you measure CPU cost?
With the DWT cycle counter around each encoder call, on the same recorded data, subtracting the measured harness overhead.
What happens if a block would get bigger?
The encoder falls back to raw storage for that block and marks it in the header.