Coverage Metrics for Scenario-Based ADAS Validation

The problem

Scenario-based validation needs a way to decide when testing is sufficient. Counting scenarios or kilometres says little, and code coverage does not reflect the traffic situations a function has seen. Several scenario coverage metrics have been proposed, but whether they correlate with real fault-finding ability is rarely tested. This project measures that relationship.

Objectives

• Build an ODD taxonomy and parameterised scenario space for a chosen function.
• Implement several scenario coverage metrics.
• Generate test suites with different coverage levels and strategies.
• Measure fault detection with seeded faults in the function under test.
• Analyse which metrics predict fault detection and propose a combined criterion.

What you need

Key concepts

ODD taxonomy

A structured list of the attributes that describe the operating environment, such as road type, junctions, weather, lighting and traffic participants, as set out in ODD taxonomy standards such as ISO 34503 and BSI PAS 1883.

Combinatorial (t-wise) coverage

The share of all combinations of t parameter values that appear in at least one test. Pairwise coverage (t = 2) is a common target because many faults involve two interacting factors.

Situation coverage

Coverage of distinct traffic situations, such as relative positions and motions of ego and others, rather than of input parameter values.

Mutation score

The share of deliberately injected faults that a test suite detects; used here as an independent measure of fault-finding ability.

Step-by-step guide

1. 1

   Scope the function and ODD

   Choose a function under test that runs in esmini and CARLA, such as a Python ACC with lane keeping or a highway pilot built from simple controllers. Build an ODD taxonomy for it using the structure of a published taxonomy standard: road geometry, lane count, junction types, speed limits, weather and lighting, and traffic participant types and behaviours. Mark which attributes are inside the ODD, at its boundary and outside it.

   Checkpoint: An ODD taxonomy table with in, boundary and out values for each attribute.

2. 2

   Define the scenario space

   Define four to six logical scenarios, such as following, cut-in, cut-out, merge and curve approach, each with parameters derived from the ODD taxonomy and kinematic parameters like speeds, gaps and timings. Discretise each parameter into meaningful classes for combinatorial metrics and keep continuous ranges for space-filling metrics. Write the logical scenarios as OpenSCENARIO templates with parameter declarations and check them in esmini.

   Checkpoint: OpenSCENARIO templates for each logical scenario that run in esmini with any valid parameter set.

3. 3

   Implement coverage metrics

   Implement at least five metrics: parameter-class coverage, pairwise and three-wise combinatorial coverage, space-filling coverage such as discrepancy or the share of grid cells occupied, ODD attribute coverage including boundary values, and situation coverage computed from simulated trajectories by classifying relative positions and motions over time. Unit-test each metric on small hand-built suites with known coverage values. Document each metric's formula and assumptions so the definitions can be reviewed independently.

   Checkpoint: Each metric returns the hand-calculated value on test suites designed for it.

4. 4

   Seed faults in the function

   Create 30 to 60 mutants of the function under test, using operators such as changed thresholds, swapped comparisons, removed limit checks, added delays and wrong lane assignments. Remove mutants that cannot change behaviour in any scenario of the space, found by running a large reference suite, so the mutation score is not distorted. Keep each mutant small and record its type and location.

   Checkpoint: A set of verified mutants, each shown to fail at least one scenario in the reference suite.

5. 5

   Generate test suites

   Generate many suites at several sizes using different strategies: random sampling, Latin hypercube, pairwise and three-wise covering arrays, ODD-boundary-focused sampling and criticality-guided search. Each suite is a list of concrete scenarios. Use different seeds so each strategy and size gives a distribution of suites rather than a single one. Store every suite with its generation settings. Check that suites of the same size from different strategies really differ in coverage, otherwise the later comparison has nothing to separate.

   Checkpoint: Hundreds of stored suites across strategies and sizes, each reproducible from its settings.

6. 6

   Run suites on the original and mutants

   Run every scenario on the original function and every mutant in esmini, with pass criteria such as no collision, minimum TTC and lane deviation limits. Cache results per scenario and mutant so suites that share scenarios are cheap to evaluate. A mutant is detected by a suite if any of its scenarios fails on the mutant but passes on the original. Compute coverage and mutation score for every suite.

   Checkpoint: A table with every suite's coverage values under each metric and its mutation score.

7. 7

   Analyse coverage against fault detection

   For each metric, compute rank correlation with mutation score across suites, controlling for suite size, since larger suites tend to score higher on both. Use partial correlation or regression with size as a covariate. Check which metrics stay predictive at fixed size. Break the analysis down by mutant type to see whether some metrics catch threshold faults while others catch timing faults.

   Checkpoint: Size-controlled correlations between each metric and mutation score, overall and by mutant type.

8. 8

   Propose and test a combined criterion

   Fit a model, such as a regularised regression, that predicts mutation score from several coverage metrics, and derive a simple combined criterion, for example pairwise coverage plus boundary coverage plus situation coverage above set levels. Test it on held-out suites and on a second function under test, so the criterion is not tuned to one system. Compare its predictive power with each single metric.

   Checkpoint: Held-out prediction error of the combined criterion compared with single metrics, on two functions.

9. 9

   Check with higher-fidelity simulation

   Repeat a subset of the study in CARLA with ScenarioRunner, using the same OpenSCENARIO templates. Check whether the ranking of metrics by predictive power holds when vehicle dynamics and sensors are more realistic. Report the cost difference between esmini and CARLA and discuss when a fast simulator is enough for coverage studies. Note any metric whose ranking changes between simulators.

   Checkpoint: A comparison of metric rankings between esmini and CARLA on the subset.

10. 10

    Write the thesis chapters

    Document the ODD taxonomy, scenario space, metric definitions, mutant design, suite generation, results and the combined criterion. Discuss threats to validity: mutants as stand-ins for real faults, a limited scenario space, simulator simplifications and the choice of pass criteria. Release the metrics as a small Python library with the suites and results so others can reuse them. Include worked examples showing how to compute each metric for a new suite.

    Checkpoint: Thesis chapters and a released metrics library with data that reproduces every figure.

Testing & validation

• Unit tests of each coverage metric on hand-built suites.
• Mutant validity checked against a large reference suite.
• Size-controlled correlation of each metric with mutation score.
• Held-out and second-system testing of the combined criterion.
• Consistency of conclusions between esmini and CARLA.

Expected results

A set of implemented scenario coverage metrics, quantitative evidence of how well each predicts fault detection, and a combined coverage criterion tested on more than one function.

Take it further

• Add real-world driving data to measure ODD coverage of recorded logs.
• Include perception faults and sensor-level scenarios.
• Use coverage gaps to drive scenario generation automatically.
• Study how coverage criteria fit into a safety case structure.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.