Camera–Radar Fusion for Forward Collision Warning

The problem

Cameras classify objects well and measure lateral position accurately but estimate range poorly; radar measures range and relative speed well but has coarse lateral resolution and can produce ghost targets. A forward collision warning built on one sensor alone either warns late or warns falsely. Fusing the two is the standard industry approach, and this project shows why on repeatable simulated scenarios.

Objectives

• Build highway and urban driving scenarios with a lead vehicle, cut-ins and stationary objects.
• Configure camera and radar sensor models with realistic accuracy and field of view.
• Fuse their detections with a multi-object tracker and select the most important object in the ego lane.
• Compute time-to-collision and issue staged warnings.
• Compare warning timing and false alarms for camera-only, radar-only and fused versions.

What you need

Key concepts

Complementary sensors

Radar gives accurate range and range-rate but poor angle; a camera gives accurate angle and object class but poor range. Fusion combines their strengths in one estimate.

Multi-object tracker

Software that associates detections from several sensors with tracks, updates each track with a filter, and manages track confirmation and deletion.

Most important object (MIO)

The closest confirmed track inside the ego lane ahead. Forward collision warning reacts only to this object, so lane assignment matters as much as tracking accuracy.

Time-to-collision

Range divided by closing speed, assuming both speeds stay constant. A warning is raised when TTC falls below a threshold that leaves time for the driver to react.

Step-by-step guide

1. 1

   Set requirements and operating domain

   Define the function: warn the driver of an imminent frontal collision with a vehicle in the ego lane at ego speeds from 30 to 100 km/h, on dry, marked roads. Set targets for warning time (for example at least 2 s before the latest braking point), false warnings per hour of driving and missed warnings. List exclusions honestly, such as pedestrians, sharp curves and heavy rain, which a simple version does not handle.

   Checkpoint: A requirement table with ODD, warning timing target and false-alarm target.

2. 2

   Build test scenarios

   In Driving Scenario Designer, create at least six scenarios: steady following, lead vehicle braking hard, a slow vehicle in the ego lane, a vehicle in the adjacent lane on a curve, a cut-in at short range and a stationary vehicle ahead. Use a two- or three-lane road with lane markings. Export each to a MATLAB function so it can be regenerated and varied by script. Record the ground-truth moment at which a warning should first be raised in each scenario.

   Checkpoint: Six scenario files that play back in the designer, with expected warning times written down.

3. 3

   Configure the sensors

   Mount a forward camera with visionDetectionGenerator on the windscreen and a long-range radar with drivingRadarDataGenerator on the front bumper. Set fields of view, maximum range and noise to plausible values and document them. Make the camera's range error grow with distance and give the radar a coarse azimuth resolution so the trade-off is visible. Also enable lane boundary detections from the camera for lane assignment.

   Checkpoint: A bird's-eye plot showing both sensors' coverage areas and detections on each scenario.

4. 4

   Collect and inspect detections

   Run each scenario and store detections as objectDetection arrays with time stamps and sensor indices. Plot detection positions against ground truth to see each sensor's error pattern. Confirm that radar ghost or clutter detections appear near guard rails if enabled, and that the camera misses objects at long range. Quantify per-sensor range and lateral position error before fusing, so the fusion benefit can be measured later.

   Checkpoint: Per-sensor error statistics for range and lateral position against ground truth.

5. 5

   Set up the multi-object tracker

   Create a multiObjectTracker with an initialisation function based on a constant-velocity extended Kalman filter (initcvekf), an assignment threshold and a confirmation threshold such as two hits in three updates. Feed detections from both sensors at their own rates. Tune the measurement noise of each sensor in the detection objects rather than in the tracker, so each sensor's strengths are respected.

   Checkpoint: Confirmed tracks follow every true vehicle on the bird's-eye plot without frequent drops or duplicates.

6. 6

   Select the most important object

   Use the camera's lane boundary detections to fit the ego lane ahead, then mark as MIO the closest confirmed track whose lateral position falls inside the lane at its range. Handle curves by evaluating the lane boundary at the track's longitudinal distance rather than assuming a straight lane. Colour the MIO on the bird's-eye plot so mistakes are obvious during playback.

   Checkpoint: In the curve scenario, the adjacent-lane vehicle is never chosen as MIO.

7. 7

   Compute TTC and warning logic

   Compute TTC from the MIO's range and range-rate. Add a simple enhanced TTC using relative acceleration if available. Raise a caution when TTC falls below a first threshold and a warning below a second, with hysteresis so the alert does not flicker. Suppress warnings when the closing speed is negative or tiny. Log the warning state against time for every scenario.

   Checkpoint: Warning timelines that rise before the expected warning point in the braking and stationary-vehicle scenarios.

8. 8

   Build single-sensor baselines

   Run the same chain twice more, once with only camera detections and once with only radar detections, keeping all other settings the same. With radar only, lane assignment has to rely on radar lateral position, which is coarse at range; with camera only, TTC is noisy because range and range-rate are poor. Record both versions' warnings for every scenario. Keep the warning thresholds identical in all three versions, so differences come only from the sensors and not from retuning.

   Checkpoint: Warning timelines for camera-only, radar-only and fused chains on the same scenarios.

9. 9

   Evaluate with variations

   Script variations of each scenario: ego speed in 10 km/h steps, lead deceleration from 2 to 6 m/s², and cut-in distances. For each run compute warning lead time against the expected point, missed warnings and false warnings. Summarise each version's results in a table and a plot of warning lead time against ego speed. Discuss where fusion helps most and where it still fails.

   Checkpoint: A results table and plot comparing the three versions over all scenario variations.

10. 10

    Report

    Document the requirement, scenarios, sensor settings, tracker configuration, MIO logic, warning logic and results. Explain each failure seen, linking it to a sensor limitation or a design choice. Note that this is a simulation study and that real FCW validation uses recorded data and test tracks, never public-road experiments. Package the scenario files and scripts so the whole study can be rerun.

    Checkpoint: A report with reproducible results and a clear statement of what the fused design improves.

Testing & validation

• Per-sensor detection error against ground truth.
• Track continuity on all scenarios without duplicate tracks.
• Correct MIO selection in curve and cut-in scenarios.
• Warning lead time, missed warnings and false warnings for all three versions.
• Sweeps over ego speed and lead deceleration.

Expected results

A simulated forward collision warning chain that fuses camera and radar, with evidence across scenario variations that fusion warns earlier and more reliably than either sensor alone.

Take it further

• Add automatic emergency braking after the warning stage.
• Replay recorded detections from a public dataset such as nuScenes in place of simulated sensors.
• Build the same chain in Simulink and generate C code with Embedded Coder.
• Add pedestrians and two-wheelers to the scenarios.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.