Gesture Recognition with the IMU for Hands-Free Controls

The problem

Riders of two-wheelers and drivers should not take their eyes or hands off the controls for long to change a display mode or acknowledge an alert. Motion-based gestures are one way to give quick commands, and the same signal-processing chain is used in many vehicle and wearable systems.

Objectives

• Stream accelerometer and gyroscope data from the MPU6050 at a fixed rate.
• Record a labelled gesture dataset from team members.
• Extract time-domain features and train a small classifier on the PC.
• Implement the classifier in fixed-point C on the STM32.
• Map gestures to dashboard actions and measure accuracy and latency.

What you need

Key concepts

Gesture segmentation

Deciding where a gesture starts and ends in a continuous stream, here by detecting when motion energy rises above a resting threshold.

Feature extraction

Reducing a window of raw samples to a few numbers such as mean, variance, peak values and zero crossings per axis, which a simple classifier can separate.

Decision tree classifier

A model that classifies by a sequence of threshold comparisons on features. It converts directly to nested if-statements in C and needs no floating point.

Confusion matrix

A table of true versus predicted classes that shows which gestures are mistaken for which, more useful than a single accuracy figure.

Step-by-step guide

1. 1

   Define the gesture set and actions

   Choose four to six gestures that are clearly different: tilt left, tilt right, tilt forward, a quick shake, a double tap on the board and 'no gesture'. Map each to an action, for example next screen, previous screen, acknowledge alert or toggle units. Include 'no gesture' as a class so normal handling does not trigger commands. Decide how the board will be held, since the MPU6050 is fixed on the kit and the whole kit moves.

   Checkpoint: A written gesture list with a description, an action and a short video of each gesture being performed.

2. 2

   Set up the MPU6050 over I2C

   Turn on the DIP switches for the MPU6050, LCD and UART blocks. In CubeMX enable I2C1 at 400 kHz and a UART at 115200 baud. Read WHO_AM_I (register 0x75, expect 0x68), wake the device by writing 0 to PWR_MGMT_1 (0x6B), set the digital low-pass filter in CONFIG (0x1A) and set SMPLRT_DIV (0x19) for 100 Hz output. Select ±4 g and ±500 °/s ranges. Read the 14 bytes from 0x3B in one burst with HAL_I2C_Mem_Read.

   Checkpoint: WHO_AM_I returns 0x68 and the Z-axis reads about 1 g when the board is flat and still.

3. 3

   Stream and calibrate data

   Use a timer interrupt at 100 Hz to read the sensor, and send all six axes over UART as compact CSV. Keep the board still for ten seconds and record the gyroscope offsets, then subtract them in firmware. Write a Python logger that saves data with a label typed by the user. Check sample timing on the PC by counting samples per second.

   Checkpoint: A steady 100 samples per second at the PC, and gyroscope readings near zero when still after offset removal.

4. 4

   Record the dataset

   Each team member performs each gesture at least 30 times, with short rests between, and records long stretches of 'no gesture' that include picking up and putting down the board. Label start and end times. Several people give variation in speed and strength, which makes the classifier more robust. Keep one person's data aside as an unseen test set. Note the date, person and board orientation in a log sheet for each session.

   Checkpoint: At least 500 labelled gesture examples plus several minutes of 'no gesture' data, saved as CSV with a data dictionary.

5. 5

   Design segmentation and features

   Compute motion energy as the magnitude of the gyroscope vector minus its resting value, and start a window when it crosses a threshold for a few samples. Take a fixed window, for example 80 samples (0.8 s). For each axis compute mean, standard deviation, minimum, maximum and the sample index of the peak. Plot features for each class to see which ones separate them. Keep the feature count small so the STM32 can compute them quickly.

   Checkpoint: Scatter plots showing clear separation of most classes on two or three features.

6. 6

   Train and select a classifier

   In scikit-learn train a decision tree with limited depth (for example 5) and compare with k-nearest neighbours and logistic regression as baselines. Use cross-validation grouped by person so results reflect new users. Choose the decision tree if its accuracy is close, because it is easiest to put on the microcontroller. Scale features to integers, for example milli-g and degrees per second, so the tree thresholds are integers.

   Checkpoint: Cross-validated accuracy and a confusion matrix for each model, and a chosen tree with integer thresholds.

7. 7

   Port the classifier to the STM32

   Write a Python script that walks the trained tree and generates nested if-else C code with integer thresholds. Implement feature extraction in C on the window buffer using integer arithmetic. Test the C code first by feeding recorded windows over UART and comparing predictions with Python, sample by sample. Any mismatch usually comes from scaling or rounding. Fix these by matching the integer scaling exactly in both versions before going live.

   Checkpoint: The STM32 gives the same prediction as Python for 100% of the recorded test windows.

8. 8

   Integrate with the dashboard

   Run segmentation and classification live. On each recognised gesture, carry out its action on a simple multi-screen LCD dashboard (for example speed, battery and trip screens with dummy values), flash the RGB LED and give a short beep. Add a confidence rule: ignore gestures that arrive within 0.5 s of the previous one. Log every detection over UART. Keep the classifier in its own module so it can be replaced later.

   Checkpoint: Each gesture changes the dashboard as designed during a live demo.

9. 9

   Test accuracy and latency

   Ask the held-out team member, and one or two people who did not record data, to perform each gesture 20 times. Count correct, wrong and missed detections and the false triggers during a few minutes of normal handling. Measure latency from gesture end to LCD change by toggling a GPIO and timing it, or by recording video. Measure the execution time of feature extraction and classification with a timer.

   Checkpoint: A confusion matrix for live tests, false triggers per minute, and latency below about 300 ms after the window ends.

10. 10

    Document and demonstrate

    Write the report covering sensor setup, dataset, features, model selection, C implementation and test results. Discuss limits: the gestures are made by moving the whole kit, and vehicle vibration in a real application would need further filtering and testing. Prepare a live demo and keep the dataset and scripts in the submission. Include the confusion matrices, timing figures, memory use from the build output and a short user guide explaining how each gesture should be made.

    Checkpoint: A report, a repository with firmware and Python scripts, and a working live demo.

Testing & validation

• WHO_AM_I and still-board readings confirm the sensor is configured correctly.
• Bit-exact agreement between Python and STM32 predictions on recorded windows.
• Live accuracy on unseen users reported with a confusion matrix.
• False triggers per minute and end-to-end latency measured.

Expected results

A working gesture-controlled dashboard on the kit with an on-device classifier, a labelled dataset and measured accuracy, false-trigger rate and latency.

Take it further

• Replace the decision tree with a small fixed-point neural network and compare.
• Use the MPU6050 FIFO and motion interrupt to save power.
• Send recognised commands over CAN to another node.
• Add an HC-05 Bluetooth module on the Bluetooth header to send gestures to a phone.

Viva and review questions