The problem
Learning-based energy management is widely published, but many studies test the agent on the same cycle it trained on, which says little about real use. The central research question is generalisation: does a learned policy stay efficient and charge-sustaining on new cycles, traffic conditions and parameter changes, and how does it compare with well-tuned classical methods under the same conditions?
Objectives
- Build a fast, validated Simulink hybrid-powertrain environment for training.
- Design state, action and reward for a charge-sustaining energy manager.
- Train DDPG or TD3 and DQN agents on a training set of cycles.
- Evaluate on held-out cycles against adaptive ECMS and DP bounds.
- Analyse generalisation, robustness to parameter change and policy behaviour.
What you need
Tools and toolboxes
- Simulink — Hybrid powertrain environment model and the RL Agent block.
- Reinforcement Learning Toolbox — Environment interface, DDPG, TD3 and DQN agents, training and simulation.
- Deep Learning Toolbox — Actor and critic network definitions used by the RL agents.
- MATLAB — Drive-cycle sets, DP and ECMS baselines, evaluation scripts and statistics.
- Parallel Computing Toolbox (optional) — Parallel training workers; training also runs serially, more slowly.
Software
- MATLAB R2023b or later
- Simulink
- Reinforcement Learning Toolbox
- Deep Learning Toolbox
- Drive-cycle library (IDC, MIDC, WLTP, NEDC, FTP-75 and recorded or synthetic cycles)
Extra parts and data
None beyond the software above.
Key concepts
- Markov decision process
- The energy manager observes a state (SOC, demand power, speed), chooses an action (fuel-cell or supercapacitor power) and receives a reward. The goal is to maximise long-run reward.
- Reward shaping
- The reward combines hydrogen use or losses with an SOC penalty. Its weights decide whether the agent is charge-sustaining, so they must be chosen and reported carefully.
- Generalisation gap
- The difference in performance between training and held-out cycles. A large gap means the agent has memorised its training cycles.
- Actor–critic methods
- DDPG and TD3 learn a deterministic policy (actor) and a value estimate (critic) for continuous actions; TD3 adds twin critics and delayed updates to reduce over-estimation.
Step-by-step guide
- 1
Frame the research question and protocol
State the question: does an RL energy manager trained on a set of cycles match adaptive ECMS on held-out cycles? Split cycles into training (for example IDC, NEDC and synthetic cycles), validation and test (MIDC, WLTP, FTP-75 and recorded city data if available). Fix metrics: SOC-corrected hydrogen or loss, gap to DP, final SOC error, constraint violations. Commit to the split before training.
Checkpoint: A written protocol with the cycle split, metrics and baselines, dated before any training run.
- 2
Build a fast environment
Use a quasi-static hybrid model at a 1 s step: vehicle load from speed, fuel-cell hydrogen map from published data, battery as OCV and resistance. Avoid Simscape here; training needs many thousands of episodes, so the model must run far faster than real time. Verify it against the slower reference model from earlier work on a full cycle. Profile the model and remove any block that dominates run time, because training speed sets how many configurations and seeds the study can afford.
Checkpoint: The fast model reproduces the reference model's hydrogen and SOC within 1 per cent and runs a full cycle in a fraction of a second.
- 3
Compute baselines first
Solve DP on every cycle for the lower bound. Implement adaptive ECMS with gains tuned on the training cycles only. Run both on all cycles and store results. Having baselines fixed before training prevents unconscious tuning of the agent to beat them on particular cycles. Store baseline results in a versioned file with the date and settings, so the comparison later is against exactly the numbers fixed before training, and keep the DP grid settings with them.
Checkpoint: A baseline table for all cycles, with ECMS tuned on training cycles only.
- 4
Design the MDP
Observations: SOC, demand power, speed, recent average demand and previous action, all normalised. Action: fuel-cell power fraction (continuous for DDPG or TD3, discretised into 11 levels for DQN). Reward: −(hydrogen flow + λ·(SOC − SOC_ref)² + μ·action change²). Episodes are one cycle with a random start SOC within a band. Create the environment with rlSimulinkEnv and an RL Agent block.
Checkpoint: A random agent runs complete episodes without errors and reward values are of order one after normalisation.
- 5
Train agents
Create TD3 and DQN agents with rlTD3Agent and rlDQNAgent, small actor and critic networks (two hidden layers of 64 to 128 units) and sensible exploration noise. Train with rlTrainingOptions, sampling a training cycle at random each episode. Use several random seeds per configuration, at least three. Save agents at regular intervals and select by validation-set performance, not training reward. Record wall-clock time per training run and the number of episodes, since compute cost is part of a fair comparison with methods that need no training.
Checkpoint: Training curves that rise and plateau for every seed, with saved checkpoints and validation scores.
- 6
Check charge sustenance and constraints
Simulate the selected agents on validation cycles and plot SOC. If SOC drifts, revisit λ in the reward; report every change and re-run all seeds. Check that fuel-cell power stays within bounds and slew limits, which should be enforced by saturation in the environment rather than learned. Keep a log of every reward change with its date and reason, because reviewers will want to know how much tuning went into the reward, and report the final SOC error for every seed.
Checkpoint: Final SOC within a stated tolerance on all validation cycles for every seed.
- 7
Evaluate on held-out cycles
Run the selected agents, ECMS and DP on test cycles. Report SOC-corrected consumption, gap to DP, final SOC error and power-rate RMS, as mean and spread over seeds. Compare training and test gaps to quantify generalisation. Use paired comparisons per cycle, since cycles differ greatly in difficulty. Do not reselect agents or change hyperparameters after seeing test results; if changes are needed, report them as a separate experiment with a fresh test split.
Checkpoint: A results table with mean ± spread across seeds for each test cycle and method.
- 8
Stress-test robustness
Change parameters the agent never saw: vehicle mass +20 per cent, battery resistance doubled to mimic ageing, and a fuel-cell efficiency curve shifted down. Re-run all methods without retraining. This shows whether the learned policy degrades more or less gracefully than ECMS, which uses an explicit model. Run each perturbation on every test cycle and every seed, and report whether any method breaks charge sustenance or violates constraints under the changed parameters.
Checkpoint: A robustness table showing the change in gap to DP for each method under each perturbation.
- 9
Interpret the policy
Plot the agent's action against SOC and demand power on a grid of inputs, and compare with the DP policy map and the ECMS decision surface. Identify regions where the agent behaves differently, for example near SOC limits. Explanations of this kind help reviewers trust or question the result. Hold the other observations at typical values when plotting each map, and note that a smooth policy map is easier to trust and certify than an irregular one.
Checkpoint: Policy-map plots for agent, ECMS and DP, with differences described.
- 10
Ablations
Remove the recent-demand observation, change network size and reduce the number of training cycles. Measure the effect on test performance. Ablations show which design choices matter and guard against over-claiming. Keep compute budgets equal between variants. Train each ablated variant with the same number of seeds and episodes as the full agent, evaluate it on the same test cycles, and report the change in gap to DP with its spread, so that small differences are not mistaken for real effects.
Checkpoint: An ablation table with the effect of each change on test-set gap to DP.
- 11
Write up
Present the protocol, environment validation, MDP design, training details (seeds, hyperparameters, compute), baselines, test and robustness results with spread, policy interpretation and ablations. State the limitations: quasi-static model, simulated cycles and no hardware validation. Release scripts, trained agents and the cycle split. Include the cycle split, all hyperparameters and reward weights, so a reader can reproduce the training. Report negative results, such as configurations that failed to train, with the same care as successes.
Checkpoint: A thesis chapter or paper draft with full reproducibility details.
Testing & validation
- Fast environment verified against a reference model.
- Baselines fixed before training, with ECMS tuned only on training cycles.
- Charge sustenance on validation cycles for every seed.
- Held-out test results reported as mean and spread across seeds.
- Robustness to unseen parameter changes compared across methods.
Expected results
A trained RL energy manager with a documented generalisation study: test-set consumption, gap to DP and robustness compared with adaptive ECMS across seeds, held-out cycles and parameter perturbations, plus policy interpretation and ablations.
Research angle for your PhD
Research questions: do learned energy managers generalise to unseen cycles and parameter changes as well as adaptive ECMS, and which observation and reward choices control that generalisation? Metrics: SOC-corrected consumption, gap to DP, final SOC error, power-rate RMS, all as mean and spread over seeds, on a pre-declared train/validation/test split. Baselines: DP bound, adaptive ECMS and a rule-based strategy. Publishable contributions include a rigorous held-out evaluation, a method that measurably narrows the generalisation gap, or evidence that a well-tuned classical method matches RL under fair comparison; negative results reported carefully are also valuable.
Take it further
- Add a speed preview from route information to the observations.
- Use domain randomisation of vehicle parameters during training.
- Combine RL with ECMS by learning the equivalence factor.
- Deploy the policy in a real-time simulation with generated code.
Viva and review questions
Why did you fix the cycle split before training?
Choosing test cycles after seeing results would let selection bias inflate performance; fixing it first makes the held-out comparison honest.
Why multiple seeds?
RL training results vary considerably with the random seed, so a single run cannot distinguish a real improvement from luck; mean and spread over seeds can.
Why enforce limits in the environment rather than through reward?
Hard limits such as maximum power and slew rate must never be broken, so saturating them in the model guarantees this, while reward only discourages violations.
What would make you prefer ECMS over the RL agent?
If ECMS achieved similar consumption on held-out cycles and degraded less under parameter changes, its simplicity and explainability would make it the better engineering choice.
Learn it at DIYguru
These DIYguru courses and programmes teach what this project needs.