Dynamic Programming Benchmark for Hybrid Storage Energy Management

The problem

Real-time energy-management strategies are usually judged by comparison with each other, which says little about how much improvement is still possible. Dynamic programming over a known cycle gives the global optimum for a chosen cost, so it measures the gap. Building a correct, efficient DP is itself a careful piece of work: discretisation, infeasible states and interpolation all affect the answer.

Objectives

• Formulate the hybrid-storage power split as a discrete-time optimal control problem.
• Implement backward DP with state-grid interpolation in vectorised MATLAB.
• Verify the solver on a small problem with a known solution.
• Benchmark rule-based and low-pass filter strategies against the DP optimum.
• Quantify the effect of grid resolution on cost and run time.

What you need

Key concepts

Bellman's principle

The optimal cost-to-go from any state depends only on that state and future decisions, so the problem can be solved backwards in time one step at a time.

State and control grids

The continuous state (supercapacitor SOC) and control (supercapacitor power) are discretised. Coarser grids run faster but give a less accurate optimum.

Infeasible states

Some state–control pairs violate limits, for example SOC leaving its window. They are given a very large cost, and the solver must handle that cost carefully at grid boundaries.

Benchmark gap

The percentage by which a real-time strategy's cost exceeds the DP optimum on the same cycle and cost function.

Step-by-step guide

1. 1

   Fix the problem definition

   Use the example semi-active system of a 48 V battery and a 58 F supercapacitor bank behind a bidirectional converter, with load power from a vehicle model on IDC at 1 s steps. Choose the cost: battery I²R loss plus converter loss, or battery RMS current squared. State the constraints: supercapacitor voltage between 50 and 100 per cent, converter power limit and battery current limit. Fix the final-state condition: end SOC equal to start SOC.

   Checkpoint: A written problem statement with state, control, dynamics, cost, constraints and terminal condition.

2. 2

   Write the one-step model

   Write a vectorised function stepModel(x, u, Pload, p) that returns next supercapacitor SOC, stage cost and a feasibility flag for arrays of states and controls at once. Supercapacitor energy changes by −u·Δt, with converter efficiency applied by direction. Battery power is Pload minus converter bus-side power; battery current follows from the quadratic of OCV, resistance and power. Return infinite cost where the battery equation has no real solution.

   Checkpoint: Unit tests: zero control keeps SOC constant, and the function returns the same numbers as a scalar loop for random inputs.

3. 3

   Implement backward DP

   Create an SOC grid of, say, 101 points and a control grid of 61 points. Initialise the terminal cost-to-go as zero at the target SOC and a large penalty elsewhere. Step backwards: for every grid state and control, compute next state, look up cost-to-go at that state with interp1 (linear), add stage cost and take the minimum over controls. Store the optimal control index for each time and state.

   Checkpoint: The solver completes on IDC in a reasonable time and the cost-to-go array contains no NaN values.

4. 4

   Handle boundaries carefully

   Interpolating between a feasible and an infeasible grid point blends a large penalty into the cost and can make good controls look bad. Use a finite penalty rather than Inf, and clamp next-state values just inside the grid, or use the boundary-line method that computes exact reachable limits per step. Compare results with and without your boundary handling to show it matters.

   Checkpoint: A plot of optimal SOC that rides close to the lower limit where useful, without artificial gaps near the boundary.

5. 5

   Run forward and verify

   Starting from the initial SOC, apply the stored optimal controls forward in time, interpolating the policy at off-grid states, and simulate with the exact one-step model. Check that the forward cost is close to the cost-to-go at the start and that the terminal SOC condition is met. Then verify on a tiny problem, a few time steps with a known optimal answer, solved by brute force.

   Checkpoint: Brute force and DP agree exactly on the tiny problem, and forward cost matches the backward value within a small percentage on IDC.

6. 6

   Study grid resolution

   Repeat the solve with SOC grids of 26, 51, 101 and 201 points and control grids of 31, 61 and 121 points. Record optimal cost and run time for each. Plot cost against grid size and fit a trend; the cost should converge as the grid is refined. Choose the coarsest grid within a stated tolerance of the finest as the working grid.

   Checkpoint: A convergence plot with the working grid marked and its cost within, for example, 0.5 per cent of the finest grid.

7. 7

   Implement real-time strategies

   Build two strategies in the same model. A low-pass filter split sends filtered load to the battery with a time constant τ and an SOC restoring term. A rule-based strategy uses supercapacitor for all regeneration and for load above a threshold, with SOC limits. Tune each reasonably on IDC so the comparison is fair, and fix the tuning before testing on other cycles.

   Checkpoint: Both strategies run charge-sustaining on IDC with their tuned parameters recorded.

8. 8

   Benchmark on several cycles

   Run DP and both strategies on IDC, MIDC and WLTP. Compute total cost, battery RMS current and peak current, and the gap of each strategy to DP. Plot supercapacitor SOC trajectories together; DP typically pre-charges before accelerations, which the real-time strategies cannot do without knowledge of the future. Present gaps as percentages of the DP cost, and keep the real-time strategies' tuning fixed from IDC so results on MIDC and WLTP show how well each strategy transfers.

   Checkpoint: A benchmark table with gaps per cycle and an SOC-trajectory comparison plot.

9. 9

   Cross-check in Simulink

   Feed the DP optimal supercapacitor power as a timeseries into the Simulink storage model used for the real-time strategies and confirm that cost matches the MATLAB forward simulation. This catches mismatches between the DP's one-step model and the simulation model, a common source of misleading benchmarks. If the costs differ, compare the two models step by step on the first few hundred seconds to locate the source of the mismatch before trusting any benchmark number.

   Checkpoint: Simulink cost matches the MATLAB forward cost within 1 per cent.

10. 10

    Extract rules from the optimum

    Plot DP supercapacitor power against load power and SOC as a scatter or binned map. Look for structure that a better real-time rule could copy, such as an SOC-dependent threshold. Implement one improved rule and measure how much of the gap it closes on a cycle it was not tuned on. Keep the improved rule simple enough to run on a microcontroller, and report its performance on every cycle, including any cycle where it does worse than the original rule.

    Checkpoint: An improved rule that reduces the gap on at least one held-out cycle, with the result reported either way.

11. 11

    Write up

    Document the formulation, solver, boundary handling, verification, grid study, benchmarks and rule extraction. Report run times and memory use. Be clear that DP needs the whole cycle in advance and is a benchmark, not a controller. Release scripts so the benchmark can be reused by later students for other strategies. Include the grid-convergence plot and the boundary-handling comparison, since reviewers will ask how you know the DP result is accurate. State the hardware and MATLAB version used for the timing figures.

    Checkpoint: A thesis chapter with reproducible scripts and a benchmark table.

Testing & validation

• Vectorised one-step model matches a scalar reference.
• DP matches brute force on a small problem.
• Forward-simulated cost matches backward value function.
• Grid-convergence study shows a stable optimum.
• Simulink re-simulation reproduces DP cost.

Expected results

A verified, vectorised DP benchmark for battery–supercapacitor energy management, with grid-convergence evidence and quantified gaps for two real-time strategies on three drive cycles, plus an improved rule derived from the optimum.

Take it further

• Add battery temperature as a second state and study the cost of a 2-D grid.
• Use DP results to train a regression policy.
• Apply the same DP framework to a fuel-cell hybrid.
• Compare with model predictive control using a short preview.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.