Perception Robustness under Rain, Fog, Night and Glare

The problem

ADAS perception trained mostly on clear daytime images can fail silently in adverse conditions, which ISO 21448 (SOTIF) treats as triggering conditions that must be identified and handled. Robustness is often reported as one number on synthetic noise, which says little about real rain or glare. A rigorous benchmark and fair comparison of remedies is needed.

Objectives

• Build a benchmark of real and synthetic adverse-condition test sets with severity levels.
• Measure detector degradation per condition and per object class.
• Validate synthetic corruptions against real adverse data.
• Compare remedies such as augmentation, domain adaptation and image restoration.
• Relate results to SOTIF triggering conditions and an ODD definition.

What you need

Key concepts

Triggering condition

In SOTIF, a specific situation such as heavy rain or low sun that can cause a functional insufficiency in a correctly working system.

Corruption severity

A graded strength of a synthetic effect, such as fog density, used to plot performance against severity rather than a single point.

Domain adaptation

Methods that adapt a model trained on one data distribution to another, for example clear-to-fog, often without labels in the target domain.

Relative robustness

Performance under a condition divided by clean performance, which allows comparison between models with different clean accuracy.

Step-by-step guide

1. 1

   Frame the questions and ODD

   State the research questions: how much does detection degrade per condition, do synthetic corruptions predict real degradation, and which remedy gives the best robustness without hurting clear-weather accuracy? Define the ODD and list the conditions of interest as SOTIF triggering conditions: rain, fog, night, low-sun glare and wet-road reflections. Write a pre-registered experiment matrix with models, conditions, seeds and metrics.

   Checkpoint: Research questions, ODD and an experiment matrix agreed with the supervisor.

2. 2

   Build the real-condition test sets

   Use BDD100K image attributes to build test subsets for clear day, rainy, foggy, snowy, night and dawn/dusk, and select rain and night scenes from nuScenes. Keep class definitions consistent across datasets with a mapping table. Check sample images by eye because weather tags can be noisy, and document how many images per condition survive the check. These subsets are never used for training.

   Checkpoint: Verified real-condition test sets with counts per condition and class.

3. 3

   Build calibrated synthetic corruptions

   Implement fog with an atmospheric scattering model using depth (from Cityscapes disparity or a monocular depth estimate), rain streaks with motion-blurred line overlays and droplets, low light with gamma, gain and sensor noise, and glare with a bright source and bloom. Generate five severity levels each from clear images. Calibrate severity by comparing image statistics such as contrast and dark-channel values against the real sets.

   Checkpoint: Synthetic test sets at five severities with statistics compared against real data.

4. 4

   Train baseline detectors

   Train two detectors of different design (for example a YOLO model and a two-stage Faster R-CNN from torchvision) on clear-weather BDD100K training images only, with fixed schedules and three seeds each. Record clean validation mAP. These are the baselines whose degradation the benchmark measures, so training settings must be identical across later remedy experiments. Save the exact data lists, configuration files and random seeds for each run, because every remedy later trained must be compared against these baselines under matching conditions and budgets.

   Checkpoint: Baseline checkpoints with clean mAP and seed variance.

5. 5

   Measure degradation

   Evaluate each baseline on every real and synthetic test set and compute mAP, relative robustness and per-class recall, especially for pedestrians and two-wheelers. Plot mAP against severity for each synthetic corruption. Use bootstrap resampling of test images to give confidence intervals, so differences between conditions and models are tested rather than eyeballed. Also compute the change in localisation quality, using mAP at stricter IoU thresholds, since fog and rain often blur box boundaries as well as reducing recall.

   Checkpoint: Degradation curves and tables with confidence intervals for all conditions.

6. 6

   Test whether synthetic predicts real

   For each condition with both real and synthetic data, compare the model ranking and the size of degradation. Compute rank correlation across models and remedies between synthetic and real results. Identify where synthetic corruptions are optimistic or pessimistic, for example fog rendered without depth, and refine the corruption models if needed. This answers whether a cheaper synthetic benchmark can stand in for real data.

   Checkpoint: Correlation results and a statement of where synthetic data is or is not a valid proxy.

7. 7

   Implement remedies

   Implement three remedy families under the same training budget: weather augmentation using the synthetic corruptions, unsupervised domain adaptation using unlabelled adverse images (for example feature alignment or pseudo-labelling), and an image restoration front-end such as a dehazing or low-light enhancement network before the detector. Keep clear-weather training data and schedules identical to the baselines. Record the extra training time and data each remedy needs, and keep the detector architecture fixed so that only the remedy itself differs from the corresponding baseline run.

   Checkpoint: Trained remedy models with logged configurations matching the baselines.

8. 8

   Compare remedies fairly

   Evaluate every remedy on all real and synthetic sets and on the clean set with three seeds. Report absolute mAP, relative robustness, change in clean accuracy and added latency for restoration front-ends. Use paired statistical tests over test images to decide whether improvements are significant. Highlight results on real adverse data, since they matter more than synthetic gains. Check that no remedy was tuned on the real adverse test sets, because that would leak test data into the design.

   Checkpoint: A comparison table with significance tests and latency costs.

9. 9

   Connect to SOTIF

   Translate the findings into SOTIF terms: list the triggering conditions where the best system still falls below an acceptable recall, describe the functional insufficiency, and propose measures such as ODD restriction, a degraded-mode warning or sensor fusion with radar. Propose a runtime monitor, for example an image-quality or visibility estimate, that detects when the system is outside its validated conditions.

   Checkpoint: A SOTIF-style table of triggering conditions, observed insufficiencies and proposed measures.

10. 10

    Write the thesis chapter and benchmark release

    Write up the benchmark construction, corruption models and calibration, degradation results, synthetic-real correlation, remedy comparison and SOTIF analysis. Release the scripts that build the benchmark from the public datasets, the corruption code and the evaluation code, without redistributing dataset images. State the limitations, including weather tag noise and the number of real adverse images. Include a short guide explaining how another group can add a new condition or a new model to the benchmark and report comparable numbers.

    Checkpoint: A chapter and a released benchmark-building toolkit that reproduces the results.

Testing & validation

• mAP and recall per condition and severity with bootstrap confidence intervals.
• Rank correlation between synthetic and real adverse results.
• Remedy comparison including clean accuracy change and latency.
• Paired significance tests across seeds and images.

Expected results

A reproducible adverse-condition benchmark built from public data, measured degradation per condition, evidence on when synthetic corruptions are a valid proxy, a fair comparison of remedies and a SOTIF-oriented summary of residual risk.

Take it further

• Add radar from nuScenes and measure how fusion restores performance.
• Use CARLA weather sweeps for controlled severity with exact labels.
• Develop and validate a runtime visibility monitor.
• Study temporal models that use previous frames to bridge glare.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.