Reinforcement Learning for EV Speed and Energy Control

The problem

Eco-driving, choosing how fast to drive and when to coast or regenerate, can save meaningful energy, but optimal control needs a model and a known route, and MPC must solve an optimisation at every step. Reinforcement learning can learn a policy from interaction that runs cheaply afterwards. Its weaknesses are training cost, sensitivity to reward design and lack of guarantees, which a careful research study must address.

Objectives

• Build a Simulink EV environment with route grade, speed limits, motor efficiency and regenerative limits.
• Design observations, actions and a reward balancing energy, trip time and constraint satisfaction.
• Train DDPG, TD3 or SAC agents and compare training stability.
• Benchmark against dynamic programming (global optimum on a known route) and MPC.
• Test generalisation to unseen routes and add safety handling for speed limits.

What you need

Key concepts

Markov decision process

The control problem stated as states, actions, transition dynamics and rewards. The agent learns a policy that maximises cumulative reward.

Actor–critic methods

DDPG, TD3 and SAC learn a policy network (actor) and a value network (critic) for continuous actions. TD3 and SAC add techniques that make training more stable than DDPG.

Reward shaping

Designing the reward so the agent learns the intended behaviour. Poor shaping can produce policies that exploit loopholes, such as driving very slowly to save energy.

Dynamic programming benchmark

On a known route with discretised states, dynamic programming finds the globally optimal speed profile. It is too slow for real-time use but gives the best achievable energy for comparison.

Step-by-step guide

1. 1

   Define the problem and protocol

   Define the task: drive an example EV along a 5–20 km route with grades and speed limits, minimising energy subject to a trip-time budget and staying below the speed limit. Define metrics: energy in Wh/km, trip time, speed-limit violations and comfort. Define the protocol: training routes, held-out test routes, number of random seeds per agent and the benchmarks. Write it down before training.

   Checkpoint: A protocol document listing routes, splits, seeds, metrics and benchmarks.

2. 2

   Build the environment

   In Simulink, model longitudinal dynamics with road load including grade from a distance-indexed lookup, a motor efficiency map and regenerative power limit, and battery energy accounting. The observation vector holds speed, distance remaining, time remaining, current and upcoming grade and speed limit over a short preview. The action is a normalised traction or braking torque. Create the environment with rlSimulinkEnv and an RL Agent block.

   Checkpoint: A random or constant policy runs episodes to completion and logs energy and time correctly.

3. 3

   Compute the dynamic programming benchmark

   For each test route, discretise distance (for example 10 m) and speed (for example 0.5 m/s) and solve backward dynamic programming in MATLAB for minimum energy with a time penalty that meets the trip budget. Check by simulating the DP speed profile through the Simulink model. This is the best achievable result on each route. Check that the DP grid is fine enough by halving the step sizes and confirming the optimal energy changes by only a small amount.

   Checkpoint: DP profiles that meet the time budget, with energy confirmed by forward simulation within a small error.

4. 4

   Design the reward

   Start with reward = −(battery energy per step) − w_t·(time penalty) − w_v·(speed-limit excess) and a terminal penalty if the trip budget is missed. Normalise terms to similar scale. Test reward variants on short training runs and inspect behaviour for loopholes such as crawling or exceeding limits on downhills. Keep a record of each reward version and the behaviour it produced. This is part of the method and should appear in the thesis.

   Checkpoint: A chosen reward version with a short note explaining each term and the loopholes it closes.

5. 5

   Train agents

   Create DDPG, TD3 and SAC agents with similar actor and critic network sizes using default architectures from the toolbox or small fully connected networks. Set sample time 0.5–1 s, episode length set by route, and use rlTrainingOptions with a stopping criterion on average reward. Train each with at least three seeds and save the agents. Fix all hyperparameters before the final runs and record them in a table. Use the same training routes and episode count for each algorithm.

   Checkpoint: Training curves for each agent and seed, with at least one algorithm converging consistently.

6. 6

   Analyse training stability

   Plot mean and spread of episode reward over seeds for each algorithm. Record wall-clock training time. Inspect failures: divergence, collapse to trivial policies or sensitivity to hyperparameters. Run a small hyperparameter study on learning rate and network size for the best algorithm. Report the number of episodes and the wall-clock time to reach a stated reward level. Note any seed that failed to converge rather than discarding it. Plot the learning curves with a shaded band for the spread across seeds.

   Checkpoint: A comparison of convergence speed and final reward with spread across seeds for each algorithm.

7. 7

   Build the MPC baseline

   Design an MPC with a linearised longitudinal model, grade preview over the horizon as a measured disturbance, speed limit as a constraint and an energy-proxy cost. Tune it on training routes. This represents a practical model-based controller that the RL agent must match or beat. Use the same vehicle model, efficiency map and speed limits as the RL environment, so differences come from the control method. Report its computation time per step.

   Checkpoint: MPC completes all routes within the time budget with no speed-limit violations.

8. 8

   Evaluate on test routes

   Run the trained agents, MPC and a simple constant-speed cruise on held-out routes. Compute energy relative to the DP optimum, time, violations and comfort. Report mean and spread across seeds. Measure inference time per step of the policy compared with MPC solve time. Plot the speed profiles of the DP optimum, MPC and best RL agent on one test route to show how their strategies differ, for example where each coasts or regenerates.

   Checkpoint: A results table with energy as a percentage above the DP optimum for each controller on each test route.

9. 9

   Add safety handling

   Add a safety layer that clips the action when speed would exceed the limit within the next step, or use constrained-reward methods. Compare violations and energy with and without the safety layer. Discuss the absence of formal guarantees in standard RL and what the safety layer does and does not ensure. Log how often the safety layer modifies the agent's action. A high rate means the agent has not learned to respect the limit and relies on the safety layer.

   Checkpoint: Zero speed-limit violations with the safety layer, with any energy cost reported.

10. 10

    Generalisation and robustness

    Test agents on routes with steeper grades, different vehicle mass and a different efficiency map than seen in training. Use domain randomisation during retraining (random mass, grade scaling) and compare robustness. Report how performance relative to DP degrades outside the training distribution. Keep the out-of-distribution test set fixed and separate from any training data. Report performance with the same metrics as the main test, relative to the DP optimum on each route.

    Checkpoint: A robustness table showing performance on out-of-distribution routes, with and without domain randomisation.

11. 11

    Write the thesis chapter or paper

    Document the environment, reward design, training protocol, results against DP and MPC, safety handling and generalisation. Release environment, training scripts, seeds and route data for reproducibility. Report negative results honestly, such as algorithms that did not converge. State limitations: a longitudinal model only, simulated routes and no traffic. Include all hyperparameters, network sizes and training times in tables so another researcher can reproduce the results.

    Checkpoint: A chapter with every result reproducible, including seeds and hyperparameters.

Testing & validation

• DP benchmark confirmed by forward simulation.
• Results reported across several seeds with spread.
• Evaluation on held-out routes not used in training.
• Speed-limit violations reported for every controller, with and without the safety layer.

Expected results

A reproducible comparison of RL agents with dynamic-programming and MPC baselines for EV eco-driving, covering energy relative to the optimum, training stability, inference cost, safety handling and generalisation to unseen routes.

Take it further

• Add a lead vehicle and traffic signals to the environment.
• Combine RL with MPC, for example RL tuning MPC weights online.
• Use offline RL from logged driving data.
• Deploy the trained policy as generated code and measure execution time on an embedded target.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.