Learned Bird's-Eye-View Fusion of Camera and Radar

The problem

Camera-only 3D detection in BEV has improved quickly, but depth estimation from images is still weak at range and in poor light. Radar is cheap, robust to weather and measures velocity, but its points are sparse and noisy. Learned BEV fusion promises to combine them in one network, and there are open questions about where in the network to fuse, how to handle radar sparsity and how much benefit remains in conditions where cameras fail.

Objectives

• Reproduce a camera-only BEV detection baseline on nuScenes with documented settings.
• Design a radar branch that encodes sparse radar points into BEV features.
• Fuse camera and radar BEV features and train the fused detector.
• Evaluate with nuScenes mAP, NDS and velocity error, broken down by range, lighting and weather.
• Run ablations on fusion location, radar accumulation and radar dropout.

What you need

Key concepts

Bird's-eye-view representation

Features from all sensors are placed on a common top-down grid around the ego vehicle. It makes fusion and downstream planning simpler than fusing in image space.

Camera-to-BEV lifting

Image features are projected into BEV using predicted depth distributions or learned attention. Depth uncertainty is the main source of camera BEV error.

Radar sparsity

Automotive radar gives only tens to a few hundred points per sweep, with noisy elevation. Accumulating several sweeps and encoding velocity helps, but adds motion blur.

nuScenes detection score (NDS)

A combined metric over mAP and true-positive errors in translation, scale, orientation, velocity and attributes. It rewards good velocity estimates, where radar should help.

Step-by-step guide

1. 1

   Define the research questions and protocol

   Write precise questions: how much does radar fusion improve mAP and velocity error over a camera-only BEV baseline, and does the gain grow at long range, at night and in rain? Fix the protocol: nuScenes official train and validation splits, the official metrics and a fixed compute budget per experiment. Decide which conditions you will report separately using nuScenes scene descriptions. Agree this protocol with your guide before training anything.

   Checkpoint: A written protocol with questions, splits, metrics, condition subsets and compute budget.

2. 2

   Prepare the data

   Download nuScenes and install the devkit. Write a data loader that reads the six camera images, the five radar sensors' points and the annotations for each keyframe, transforming radar points into the ego frame with the calibrated extrinsics. Accumulate several radar sweeps with ego-motion compensation and keep each point's compensated velocity. Visualise radar points over images and in BEV with Open3D to check the transforms.

   Checkpoint: Overlay plots where radar points fall on vehicles in images and in BEV for several samples.

3. 3

   Reproduce the camera-only baseline

   Choose an open-source camera BEV detector and train it on the training split at a resolution your GPUs can handle. Use the published configuration as closely as possible and document every change. Evaluate on the validation split with the devkit. Reproducing a known baseline, even at reduced scale, is what makes your fusion results credible; note the gap to published figures and its likely causes.

   Checkpoint: Baseline mAP and NDS on the validation split, with the gap to published values explained.

4. 4

   Design the radar branch

   Encode radar points into BEV using a pillar-style encoder: features per point such as position, radar cross-section, compensated velocity and time offset, scattered into the same BEV grid as the camera features. Try a lightweight convolutional backbone on the radar BEV map. Train a radar-only detector briefly to check the branch learns something useful, especially velocity, before fusing it. Normalise each feature using training-set statistics only. Check the grid resolution and range match the camera BEV grid exactly.

   Checkpoint: A radar-only model with non-trivial mAP and lower velocity error than the camera baseline.

5. 5

   Fuse and train

   Concatenate camera and radar BEV features and pass them through a fusion convolution block before the detection head. Train the fused model with the same schedule, augmentation and compute budget as the camera baseline, so the comparison is fair. Monitor training loss and validation metrics at fixed intervals. Save checkpoints and log hyperparameters with each run so every number in the thesis can be traced to a model.

   Checkpoint: A trained fused model with validation mAP, NDS and mean velocity error logged.

6. 6

   Evaluate by condition

   Use the devkit's evaluation on the full validation split, then on subsets: distance bins (for example 0–20, 20–40 and beyond 40 m), day versus night and clear versus rain, using scene descriptions to select scenes. Report per-class results for cars, pedestrians, motorcycles and bicycles, the classes most relevant to Indian traffic. Use bootstrap resampling over scenes to give confidence intervals.

   Checkpoint: Tables of mAP, NDS and velocity error by distance, lighting, weather and class with intervals.

7. 7

   Run ablations

   Test the design choices one at a time: early fusion on radar points projected to images versus BEV-level fusion, number of accumulated radar sweeps, and removing velocity from radar features. Also train with random radar dropout and test with radar removed, to check the fused model does not collapse without radar. Keep the compute budget fixed across ablations. Where compute allows, repeat key ablations with a second random seed to show the differences are larger than run-to-run variation.

   Checkpoint: An ablation table with each change's effect on mAP, NDS and velocity error.

8. 8

   Analyse errors qualitatively

   Visualise predictions against ground truth in BEV for samples where fusion helps most and where it fails. Look for patterns: radar fixing depth errors for distant cars, radar ghosts creating false positives near barriers, or motion blur from accumulated sweeps around turning vehicles. Group errors into categories and count them on a random sample of frames, so qualitative findings have numbers.

   Checkpoint: An error taxonomy with counts on a sample of at least 200 frames.

9. 9

   Measure inference cost

   Measure inference time and GPU memory for camera-only and fused models at batch size one. Optionally export to ONNX and build a TensorRT engine at FP16 to measure latency on an embedded GPU. Report how much latency the radar branch adds compared with its accuracy gain, since an ADAS ECU has a fixed frame budget, for example 100 ms per cycle.

   Checkpoint: A latency and memory table for both models, with an accuracy-per-millisecond comparison.

10. 10

    Check transfer to Indian conditions

    nuScenes has no radar data from Indian roads, so test qualitatively what you can: run the camera branch on a sample of IDD images and inspect detections of auto-rickshaws, two-wheelers and mixed traffic. State clearly that radar fusion on Indian roads cannot be measured with these datasets. This honest discussion of domain shift is important for a thesis aimed at Indian deployment.

    Checkpoint: A short qualitative section on IDD samples with clearly stated limits on the conclusions.

11. 11

    Write the thesis chapters

    Present the protocol, baseline reproduction, radar branch, fusion design, condition-wise results, ablations, error analysis and inference cost. Make every claim traceable to a logged run. Discuss threats to validity such as reduced resolution, single dataset and seeds. Release code, configurations and trained weights where licensing permits, respecting the nuScenes dataset terms. Write the related-work chapter around the specific design choices you ablated, so the contribution is clear against published methods. Keep a run log that maps every table entry to its configuration file and checkpoint.

    Checkpoint: Thesis chapters with traceable results, ablations and a threats-to-validity section.

Testing & validation

• Reproduced camera-only baseline with explained gap to published values.
• Official nuScenes mAP, NDS and mean velocity error for camera, radar and fused models.
• Results by distance, lighting, weather and class with bootstrap intervals.
• Ablations on fusion location, sweep accumulation, velocity features and radar dropout.
• Inference latency and memory for camera-only and fused models.

Expected results

A learned camera–radar BEV detector with a carefully controlled comparison against a reproduced camera-only baseline, showing where radar helps, what it costs and where it fails.

Take it further

• Add temporal fusion of BEV features across frames.
• Use radar to supervise or refine camera depth estimation.
• Distil the fused model into a smaller network for embedded deployment.
• Study robustness to radar calibration errors.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.