The problem
Many ADAS hazards do not come from faults but from performance limits: a perception function that works as designed can still miss a pedestrian in fog or glare. ISO 21448 asks developers to find these triggering conditions and reduce the residual risk. This project carries out that analysis on a realistic perception function.
Objectives
- Define the function, its ODD and the hazardous behaviour of concern.
- Identify candidate triggering conditions from analysis and literature.
- Test the detector systematically across these conditions in CARLA.
- Confirm the most important conditions on real-world images.
- Propose and evaluate measures that reduce the identified risk.
What you need
Tools and toolboxes
- CARLA — Rendering pedestrian scenes under controlled weather, lighting and occlusion.
- Python — Scenario generation, data collection and statistical analysis.
- Ultralytics YOLO — The pedestrian detector under analysis.
- PyTorch — Running and fine-tuning the detector.
- MATLAB — Optional: statistical modelling of detection probability against conditions.
Software
- CARLA with its Python API
- Python 3 with PyTorch, Ultralytics YOLO, pandas and statsmodels
- BDD100K or a similar labelled dataset with weather and time-of-day attributes
Extra parts and data
- PC with a dedicated GPU — Rendering thousands of CARLA frames and running the detector needs a GPU.
Key concepts
- SOTIF
- Safety of the intended functionality: avoiding unreasonable risk from performance limits and foreseeable misuse, rather than from hardware or software faults.
- Triggering condition
- A specific condition of a scenario, such as low sun behind a pedestrian, that triggers a functional insufficiency and leads to hazardous behaviour.
- Known and unknown hazardous scenarios
- SOTIF aims to reduce the area of known hazardous scenarios by measures and to shrink the unknown area by exploration and testing.
- Detection probability model
- A statistical model, such as logistic regression, that predicts the chance of detection from scenario factors, showing which factors matter most.
Step-by-step guide
- 1
Define the function and hazard
Specify the function: a front camera pedestrian detector feeding an AEB, required to detect pedestrians in the ego path within 40 m. Write the ODD: urban roads, speeds up to 50 km/h, day and night, dry and rainy weather. Define the hazardous behaviour as a missed or late detection of a pedestrian in the path, and the acceptance criterion as a minimum detection rate per condition that you justify in writing.
Checkpoint: A function description, ODD and hazard definition with a stated acceptance criterion.
- 2
List candidate triggering conditions
Use a structured approach: go through the sensing chain, light, scene, object, optics and algorithm, and for each ask what could degrade detection. Candidates include low sun and glare, night with poor street lighting, fog, heavy rain, partial occlusion by parked cars, small or distant pedestrians, unusual poses, dark clothing on a dark background, and pedestrians carrying large objects. Rate each by expected impact and how often it occurs in the ODD.
Checkpoint: A table of candidate triggering conditions with a rationale and a priority for each.
- 3
Design the test space
Turn the top conditions into parameters CARLA can control: sun altitude and azimuth, cloudiness, precipitation, fog density and wetness through carla.WeatherParameters; pedestrian distance, lateral offset and walker model; occlusion fraction by a parked vehicle placed between camera and pedestrian. Choose ranges and a sampling design, for example a full grid on the main factors and random sampling on the rest, keeping the total within your rendering budget.
Checkpoint: A parameter table with ranges and a sampling plan with the total number of frames.
- 4
Generate the synthetic dataset
Write a CARLA script in synchronous mode that, for each sample, sets the weather, places the ego camera and a pedestrian, adds the occluding vehicle if needed, ticks the world and saves the image. Compute ground-truth 2D boxes by projecting the walker's bounding box into the image, and compute the visible fraction using the semantic segmentation or instance camera. Store images, boxes and all parameters in a table.
Checkpoint: A labelled synthetic dataset with every frame tagged by its parameter values, spot-checked by hand.
- 5
Evaluate the detector
Run a pretrained YOLO detector on every frame and match detections to ground truth with an IoU threshold of 0.5, counting a pedestrian as detected only above a fixed confidence threshold. Record detection, confidence and box error per frame. Compute recall per condition and per combination of conditions, and plot recall against distance for each weather and lighting setting. Keep the detector fixed during this analysis.
Checkpoint: Recall tables and plots for each condition and key combinations.
- 6
Model detection probability
Fit a logistic regression of detection against the parameters and their main interactions, for example distance times fog density, using statsmodels or MATLAB. Report coefficients with confidence intervals and identify the factors and combinations that reduce detection most. Use the model to find the region of the parameter space where predicted detection falls below the acceptance criterion. These regions are the confirmed triggering conditions in simulation.
Checkpoint: A fitted model with significant factors and a map of the parameter region below the acceptance criterion.
- 7
Confirm on real images
Check whether the simulated triggering conditions also hold in real data. Use a labelled dataset with weather and time-of-day attributes, such as BDD100K, and compare pedestrian recall across its night, rain and clear subsets, and across pedestrian box sizes as a proxy for distance. Look for the same ranking of conditions as in simulation. Note any condition that matters in one domain but not the other.
Checkpoint: A comparison of condition rankings between synthetic and real data with agreements and disagreements noted.
- 8
Propose and test measures
Propose measures for the main triggering conditions: fine-tuning on targeted data, lowering the detection threshold with temporal confirmation, restricting the ODD, or relying on a second sensor such as radar in fog. Implement at least one, for example fine-tuning on synthetic fog and night frames with a held-out test set, and re-evaluate recall in the affected region and the false-positive rate elsewhere.
Checkpoint: Before-and-after recall in the triggering-condition region with the change in false positives.
- 9
Write the SOTIF argument
Document the analysis as a SOTIF work product: function and ODD, hazard, identified triggering conditions with evidence, measures and their effect, and the remaining known limitations. State honestly what has not been covered, such as conditions CARLA cannot render well, and how unknown hazardous scenarios would be explored further. Include the dataset generator and analysis scripts so the study can be repeated for a new detector.
Checkpoint: A structured SOTIF report with traceable evidence for each triggering condition and measure.
Testing & validation
- Ground-truth boxes spot-checked against rendered images.
- Recall per condition with confidence intervals.
- Logistic model fit quality and significant factors.
- Agreement of condition rankings between synthetic and real data.
- Effect of the chosen measure on recall and false positives.
Expected results
A documented SOTIF analysis of a pedestrian detector, identifying its triggering conditions with simulation and real-data evidence, and showing the effect of at least one mitigation.
Research angle for your M.Tech thesis
Research questions: can systematic simulation reliably identify the triggering conditions of a camera detector, and do they transfer to real data? Metrics: recall per condition, significance of factors in the detection model, rank correlation of condition severity between synthetic and real datasets, and recall gain from targeted mitigation. A method that turns SOTIF analysis into repeatable, quantitative evidence is a relevant contribution for an M.Tech thesis and possible publication.
Take it further
- Add a radar model and analyse triggering conditions for camera-radar fusion.
- Use adaptive search to find triggering conditions with fewer frames.
- Repeat the analysis on IDD images for Indian traffic conditions.
- Analyse temporal effects such as flicker and track loss in video sequences.
Viva and review questions
How is SOTIF different from ISO 26262?
ISO 26262 deals with hazards caused by malfunctions of electrical and electronic systems. SOTIF deals with hazards that arise when the system works as designed but its performance is insufficient in some situations, or under foreseeable misuse.
What is a triggering condition in your project?
A specific scene condition, such as dense fog combined with a distant pedestrian, under which the detector's recall falls below the acceptance criterion.
Why confirm on real data?
Simulated rendering may not reproduce real optical effects, so a condition found in CARLA might be an artefact. Checking real data shows whether the finding holds outside the simulator.
Learn it at DIYguru
These DIYguru courses and programmes teach what this project needs.