The problem
Scenario-based testing only pays off when scenarios are written in an exchangeable format and executed automatically, so that every change to an ADAS function is checked against the same catalogue. ASAM OpenSCENARIO is the industry format for describing dynamic behaviour, and an automated pipeline around it is a core part of modern ADAS validation.
Objectives
- Write ADAS test cases in ASAM OpenSCENARIO XML, linked to requirements.
- Execute them on CARLA with ScenarioRunner in synchronous mode.
- Vary scenario parameters systematically and evaluate pass criteria automatically.
- Integrate the pipeline with version control so every change is tested.
- Measure determinism, execution time and fault-detection ability of the pipeline.
What you need
Tools and toolboxes
- CARLA — Simulation of the road, ego vehicle, traffic and sensors.
- CARLA ScenarioRunner — Executing OpenSCENARIO files on CARLA and evaluating criteria.
- ASAM OpenSCENARIO and OpenDRIVE — Describing dynamic scenarios and road networks in open formats.
- Python with pytest — Orchestrating runs, parameter variation, result checks and JUnit-style reports.
- esmini — Optional: a lightweight open-source OpenSCENARIO player for quick syntax and logic checks.
Software
- CARLA and ScenarioRunner
- Python 3 with pytest, lxml and pandas
- Git
Extra parts and data
- PC with a dedicated GPU — CARLA needs a GPU to run scenarios at a useful speed, especially when many runs are batched.
Key concepts
- OpenSCENARIO storyboard
- The structure of an OpenSCENARIO file: Init sets initial positions and speeds, then Stories contain Acts, ManeuverGroups, Maneuvers and Events with Actions triggered by Conditions.
- Logical and concrete scenarios
- A logical scenario has parameter ranges, such as cut-in distance from 5 to 25 m; a concrete scenario fixes every value and can be executed.
- Pass criteria
- Measurable checks evaluated during or after a run, such as no collision, minimum TTC above a threshold or lane departure within a limit.
- Regression testing
- Rerunning the same tests after every change, so a fix in one place that breaks something elsewhere is found immediately.
Step-by-step guide
- 1
Derive test cases from requirements
Choose a function under test, for example an ACC with stop-and-go and a lane keeping assist, implemented as a Python agent controlling the ego vehicle. Write five to eight functional requirements with IDs and derive logical scenarios from them: lead vehicle braking, cut-in, cut-out, slow vehicle on a curve and lane drift. For each logical scenario list its parameters with ranges and units, and the pass criteria with thresholds, in a traceability table.
Checkpoint: A table linking every requirement to at least one logical scenario with parameters and criteria.
- 2
Write OpenSCENARIO files
Write each logical scenario as an OpenSCENARIO XML file with a ParameterDeclarations section for its variables. Reference a CARLA town through its OpenDRIVE road network. In Init, set teleport and speed actions for the ego and other actors; in the Storyboard, define events such as a lane change action triggered when the relative distance condition is met. Validate each file against the OpenSCENARIO XML schema and play it in esmini to check the motion quickly.
Checkpoint: All scenario files pass schema validation and play back in esmini with the intended motion.
- 3
Run scenarios on CARLA
Run each file on CARLA with ScenarioRunner using its OpenSCENARIO mode, with the ego vehicle controlled by your agent rather than the default autopilot. Make sure synchronous mode is used and that sensor data reaches the agent every tick. Fix any differences between esmini and CARLA behaviour, for example lane change shapes or actor spawn heights. Record the ScenarioRunner result output for each run.
Checkpoint: Each scenario completes on CARLA with the agent in control and produces a result record.
- 4
Implement pass criteria
Use ScenarioRunner's built-in criteria such as collision tests, and add your own evaluated from logged ground truth: minimum TTC to the lead vehicle, maximum deceleration, time-headway in steady following and maximum lateral deviation from the lane centre. Write these as Python functions that take a run log and return pass, fail and a margin. Test each criterion with a deliberately failing run, such as an agent with braking disabled.
Checkpoint: Every criterion returns the correct verdict on one passing and one deliberately failing run.
- 5
Generate concrete scenarios
Write a generator that samples parameter values for each logical scenario, using a full grid for low-dimensional cases and Latin hypercube sampling for higher dimensions. For each sample, write a concrete OpenSCENARIO file by substituting parameter values, or pass them as parameter overrides if your ScenarioRunner version supports it. Store the sample table with a seed so every concrete set can be regenerated exactly.
Checkpoint: A reproducible set of a few hundred concrete scenarios generated from one seed.
- 6
Build the pipeline
Wrap execution in pytest: each concrete scenario becomes a parametrised test that starts the run, evaluates the criteria and asserts the verdict. Produce a JUnit XML report and an HTML summary. Add a script that, on every commit to the function's repository, runs a fast smoke subset and, nightly, the full set. Restart the CARLA server automatically after crashes and mark affected tests as errors rather than failures.
Checkpoint: One command runs the suite and produces a JUnit report and a summary of pass, fail and error counts.
- 7
Measure determinism and run time
Run the same concrete scenarios several times and compare trajectories and verdicts. Identify sources of non-determinism, such as asynchronous sensor callbacks, physics substepping or random Traffic Manager behaviour, and remove them with fixed seeds and synchronous settings. Measure wall-clock time per scenario and the total suite time, and record how many runs can be done per hour on your machine.
Checkpoint: Repeated runs give identical verdicts, with trajectory differences quantified, and a run-time figure per scenario.
- 8
Evaluate fault detection with mutants
Create mutants of the function under test, small deliberate faults such as a wrong sign in the gap error, a delayed brake command or a halved lateral gain. Run the suite on each mutant and record which tests detect it. Compute the mutation score, the share of mutants detected, for the grid and for Latin hypercube sampling at the same budget. Analyse which mutants survive and which scenarios would have caught them.
Checkpoint: A mutation score table per sampling method with a list of surviving mutants and why they survived.
- 9
Report
Document the requirement-to-scenario traceability, the OpenSCENARIO files, the pipeline design, determinism results, run times and mutation scores. Present a regression example: a commit that broke a requirement and the test that caught it. State the limitations: simulated sensors, a limited set of scenarios and agent behaviour simpler than production software. Provide the repository so the pipeline can be rerun. Add a short guide for writing a new test case, so the catalogue can grow beyond your project.
Checkpoint: A report and repository with a reproducible pipeline and documented results.
Testing & validation
- Schema validation of all OpenSCENARIO files.
- Each pass criterion checked against known passing and failing runs.
- Verdict determinism across repeated runs.
- Mutation score of the suite for different sampling methods.
- Suite run time and throughput.
Expected results
A working OpenSCENARIO-based regression pipeline on CARLA with traceable test cases, automatic verdicts and quantified determinism and fault-detection ability.
Research angle for your M.Tech thesis
Research questions: how should parameter space be sampled to find faults in an ADAS function at a fixed test budget, and how deterministic and transferable are OpenSCENARIO tests across simulators? Metrics: mutation score, faults found per hour of simulation, verdict agreement across repeated runs and between esmini and CARLA. Baselines: full grid and uniform random sampling. A reproducible pipeline with mutation-based evaluation gives quantitative evidence suitable for a thesis chapter or conference paper on test efficiency.
Take it further
- Add sensor-level tests with camera perception in the loop.
- Export the same scenarios to a MATLAB toolchain and compare verdicts.
- Prioritise tests by past failure rate to shorten the commit-stage suite.
- Use OpenSCENARIO parameter value distributions for stochastic sampling.
Viva and review questions
What is the difference between a logical and a concrete scenario?
A logical scenario defines parameters with ranges; a concrete scenario fixes every parameter to one value so it can be executed.
Why use mutation testing to judge a test suite?
Passing tests alone say nothing about whether the suite could find faults. Injecting known faults and counting how many the suite detects measures its fault-finding ability.
Why might the same scenario give different results in two runs?
Asynchronous timing, physics substeps or random traffic behaviour can change outcomes. Synchronous mode, fixed steps and fixed seeds remove most of these sources.
Why write scenarios in OpenSCENARIO rather than in Python directly?
OpenSCENARIO is a standard format that can be reviewed, exchanged and run in other tools, so the test catalogue is not tied to one simulator or script.
Learn it at DIYguru
These DIYguru courses and programmes teach what this project needs.