The problem
Simulation is attractive for perception testing because it is cheap, safe and labelled automatically, but a detector's performance on rendered images may not match its performance on real ones. If simulation is to be used as evidence, the size and structure of this gap must be measured, including whether simulation at least ranks conditions and models in the same order as real tests.
Objectives
- Build matched synthetic and real evaluation sets with comparable conditions.
- Measure detection performance gaps between synthetic and real data.
- Test whether simulation ranks models and conditions in the same order as real data.
- Evaluate domain randomisation, image translation and mixed training to reduce the gap.
- Propose a method to calibrate simulation-based performance estimates.
What you need
Tools and toolboxes
- CARLA — Generating labelled synthetic images with controlled weather, lighting and traffic.
- Python — Data pipelines, evaluation and statistics.
- PyTorch — Training and evaluating detectors and image translation models.
- Ultralytics YOLO — A family of detectors of different sizes for model-ranking experiments.
- OpenCV — Image processing, augmentation and visualisation.
Software
- CARLA
- Python 3 with PyTorch, Ultralytics YOLO, OpenCV, pandas and SciPy
- KITTI, BDD100K and IDD datasets under their licences
Extra parts and data
- GPU workstation or GPU cloud credits — Training several detectors and rendering large synthetic datasets needs substantial GPU time.
Key concepts
- Sim-to-real gap
- The difference between a model's performance measured on simulated data and on real data. It has an appearance part (how images look) and a content part (which objects and situations appear).
- Domain randomisation
- Training or testing with widely varied textures, lighting and colours in simulation so the model does not rely on simulator-specific appearance.
- Ranking agreement
- Whether simulation orders models or conditions the same way real data does, measured with rank correlation. Simulation can be useful for comparison even if absolute scores differ.
- Feature distribution distance
- A measure, such as the Fréchet distance between deep feature distributions, of how different two image sets look to a network.
Step-by-step guide
- 1
Define the research questions and scope
Fix the task: 2D detection of cars, pedestrians and two-wheelers from a front camera. Write the questions: how large is the gap in mAP, does it vary by condition, does simulation rank models and conditions as real data does, and which methods reduce the gap or make it predictable? Choose the real datasets, for example KITTI for clean scenes, BDD100K for weather and time of day and IDD for Indian traffic, and record their licence terms.
Checkpoint: A protocol document with questions, datasets, classes, metrics and licence notes.
- 2
Harmonise classes and conditions
Map each dataset's classes to the common set and define how ambiguous classes such as riders and autorickshaws are handled, keeping these choices fixed. Define condition labels shared across domains: day or night, clear or rain or fog, object size bins and occlusion levels. For real data, use the dataset attributes and box sizes; for synthetic data, record the exact generation parameters.
Checkpoint: A class mapping and a condition labelling scheme applied to every dataset, with counts per condition.
- 3
Generate matched synthetic data
Write CARLA generation scripts that sample towns, weather, sun position, traffic density and camera height, using a camera with intrinsics similar to each real dataset. Produce 2D boxes from projected 3D bounding boxes and filter by visibility using the instance segmentation camera. Generate a training set and condition-balanced test sets that match the real test sets' condition and object-size distributions as closely as possible.
Checkpoint: Synthetic train and test sets with condition and size histograms matched to the real test sets.
- 4
Train the model set
Train several detectors of different sizes and architectures on real training data only, with fixed recipes and seeds, so there is a set of models with genuinely different real-world performance. Train a second set on synthetic data only and a third on mixtures. Record training details for every model. This model set is what the ranking-agreement experiments rely on, so keep it large enough, at least six to eight models.
Checkpoint: A documented set of trained models with their real-data validation mAP.
- 5
Measure the absolute gap
Evaluate every model on the synthetic and real test sets with the same [email protected] and [email protected]:0.95 computation and the same class and size definitions. Report the gap per model, per class, per condition and per object size. Use bootstrap resampling to give confidence intervals on mAP and on the gap. Separate appearance from content effects by comparing condition-matched subsets. Keep the evaluation code identical across domains so differences cannot come from scoring.
Checkpoint: Gap tables with confidence intervals by model, class, condition and size.
- 6
Measure ranking agreement
For the model set, compute the rank correlation, Spearman and Kendall, between model rankings on synthetic and real test sets. Do the same for conditions: rank conditions by difficulty in each domain and correlate. Test whether simulation correctly predicts which model is better in pairwise comparisons, and with what accuracy. This answers whether simulation is useful for relative decisions even when absolute scores differ.
Checkpoint: Rank correlations for models and conditions with confidence intervals from bootstrapping.
- 7
Relate the gap to image statistics
Compute a feature distribution distance between synthetic and real test sets, using features from a fixed pretrained network, overall and per condition. Check whether conditions with larger feature distance show larger performance gaps. Include simple image statistics such as contrast, noise and colour distributions. This tells you whether the gap can be predicted from data alone before running detectors. Report which statistics, if any, add predictive value beyond the feature distance.
Checkpoint: A scatter plot and correlation between feature distance and performance gap across conditions.
- 8
Test gap-reduction methods
Evaluate methods to make synthetic data more realistic or models less sensitive: domain randomisation of textures and lighting in CARLA, image-to-image translation of synthetic frames towards real appearance with an unpaired translation network, sensor noise and blur models, and mixed synthetic and real training. Measure each method's effect on the absolute gap and on ranking agreement, since a method may reduce one without improving the other.
Checkpoint: A table of methods with their effect on absolute gap and ranking agreement.
- 9
Calibrate simulation-based estimates
Fit a calibration model that predicts real-data mAP from synthetic-data mAP and condition descriptors, trained on part of the model set and tested on the rest. Evaluate prediction error and whether prediction intervals cover the real values at the stated rate. Discuss how such calibrated estimates could be used in a validation argument and what extra real-data evidence would still be needed.
Checkpoint: Held-out prediction error and interval coverage of the calibration model.
- 10
Write the thesis and release artefacts
Write chapters on the protocol, matched data generation, gap measurement, ranking agreement, predictors of the gap, reduction methods and calibration. Discuss threats to validity: dataset label differences, imperfect condition matching, a limited model set and one simulator. Release the generation scripts, class mappings, evaluation code and trained model checkpoints where licences allow, so others can repeat the study. Give a short practical recommendation on when simulation results can and cannot stand in for real-data tests.
Checkpoint: Thesis chapters and a released code and data package that reproduces the main results.
Testing & validation
- Condition and size distributions matched between synthetic and real test sets.
- mAP gaps with bootstrap confidence intervals.
- Rank correlation of models and conditions across domains.
- Correlation between feature distance and performance gap.
- Held-out accuracy and interval coverage of the calibration model.
Expected results
A quantified, condition-resolved picture of the sim-to-real gap for camera detection, evidence on whether simulation ranks models and conditions correctly, and a tested method to calibrate simulation-based performance estimates.
Research angle for your PhD
Research questions: how large and how structured is the sim-to-real gap for camera detection, does simulation preserve the ranking of models and conditions, and can the gap be predicted and calibrated? Metrics: mAP gap with confidence intervals, rank correlation, feature distribution distance and calibration error. Much prior work reports only absolute gaps for one model; a ranking- and calibration-focused study across several models, conditions and real datasets, including Indian traffic data, is a solid basis for a doctoral contribution and publications.
Take it further
- Extend to lidar detection with CARLA lidar and real point cloud datasets.
- Study temporal metrics such as tracking stability across domains.
- Compare a second simulator or neural rendering of real scenes.
- Analyse the gap specifically for Indian traffic classes in IDD.
Viva and review questions
Why measure ranking agreement and not only absolute mAP gap?
Many engineering decisions are comparisons between models or versions. If simulation ranks them correctly it is useful for those decisions even when absolute scores are off.
How did you separate appearance effects from content effects?
By matching condition and object-size distributions between domains and comparing matched subsets, so remaining differences come mainly from how the images look rather than what is in them.
Why use several real datasets?
Each real dataset has its own gap to the others. Using several shows whether the sim-to-real gap is larger than ordinary real-to-real differences and whether findings hold across geographies.
What would you still need besides calibrated simulation results?
Real-data testing to confirm the calibration holds for the system being validated, especially in conditions poorly represented in the calibration set.
Learn it at DIYguru
These DIYguru courses and programmes teach what this project needs.