Track-to-Track Fusion of Camera and Radar Trackers

The problem

In real vehicles, camera and radar modules often come from different suppliers and each outputs its own object list, not raw detections. The central ADAS ECU must fuse tracks, not detections. Track estimates from different sensors share the same target motion, so their errors are correlated, and fusing them as if they were independent makes the result over-confident. This project studies how to fuse tracks correctly and what is lost compared with central fusion.

Objectives

• Build scenarios with camera and radar sensors, each feeding its own local tracker.
• Configure a track-to-track fuser with source configurations for both local trackers.
• Compare naive, cross-covariance and covariance intersection state fusion.
• Compare track-level fusion with central detection-level fusion on accuracy and consistency.
• Study the effect of sensor update rates, latency and dropouts on fused tracks.

What you need

Key concepts

Decentralised (track-level) fusion

Each sensor runs its own tracker and sends tracks to a fuser, which combines them. It suits supplier-built sensor modules and reduces bandwidth, but loses some information.

Correlated track errors

Two trackers following the same vehicle share the same unknown manoeuvres, so their errors are correlated. Ignoring that makes the fused covariance too small.

Covariance intersection

A fusion rule that gives a consistent result for any unknown correlation by taking a weighted combination of inverse covariances. It is conservative but safe.

Central fusion

All raw detections go to one tracker. It is the accuracy benchmark, but needs more bandwidth and access to raw sensor data.

Step-by-step guide

1. 1

   Define the architecture and requirements

   Draw the target architecture: a front camera module and a front radar module each running a local tracker, sending object lists over the vehicle network to a central fusion ECU. Set targets for fused position and velocity error, fused track consistency and the maximum number of objects. State assumed update rates, for example camera at 15 Hz and radar at 20 Hz, and a network latency. These become the experiment variables.

   Checkpoint: An architecture diagram with rates, latency assumptions and numerical targets.

2. 2

   Build scenarios and sensors

   Create highway and urban scenarios in drivingScenario with several vehicles, a cut-in and a target braking. Mount a visionDetectionGenerator and a drivingRadarDataGenerator with realistic, different error patterns: the camera accurate in angle and poor in range, the radar the reverse. Fix seeds and save detections and ground truth for each run, so all fusion variants receive identical input data. Include a vehicle partly occluded by another, so one sensor loses it while the other keeps it.

   Checkpoint: Saved detections from both sensors with per-sensor error statistics against ground truth.

3. 3

   Set up the local trackers

   Create one trackerGNN per sensor with a constant-velocity EKF from initcvekf, gate and track logic tuned for that sensor. Run each on its own detections and evaluate its tracks against ground truth with trackErrorMetrics. The camera tracker should have good lateral accuracy and the radar tracker good longitudinal accuracy; confirm this before fusing so the fusion benefit is clearly attributable.

   Checkpoint: Per-tracker accuracy tables showing the expected complementary strengths.

4. 4

   Configure the track fuser

   Create a fuserSourceConfiguration for each local tracker, with source indices and whether the tracks are in a local or central frame. Create a trackFuser with FuserIndex, the source configurations, an assignment threshold and StateFusion set to 'Cross' for cross-covariance fusion. Feed it the local tracks at each fusion time and log the fused tracks. Check that fused track IDs remain stable through the scenario.

   Checkpoint: Fused tracks appear for every true vehicle and keep their IDs through the cut-in scenario.

5. 5

   Compare state fusion rules

   Run the fuser with StateFusion set to 'Cross' (fusexcov), 'Intersection' (fusecovint) and a naive information-sum fusion you write yourself that assumes independent errors. For each, compute fused RMSE and NEES over Monte Carlo runs. The naive rule usually looks accurate but has NEES well above the consistency bound, showing it is over-confident, which is dangerous for downstream decisions. Report all three on the same Monte Carlo seeds so that the differences cannot come from the data.

   Checkpoint: RMSE and NEES for the three fusion rules, with consistency bounds drawn on the NEES plot.

6. 6

   Build the central fusion benchmark

   Feed all camera and radar detections directly into a single central trackerGNN with the same filter model. Evaluate its accuracy and consistency on the same runs. This is the upper bound for accuracy. Also count the data volume sent to the central ECU per second in both architectures, using the number of detections and tracks and a fixed size per item, to quantify the bandwidth trade-off.

   Checkpoint: Accuracy and data-volume figures for central fusion alongside track-level fusion.

7. 7

   Study rates, latency and dropouts

   Vary the camera rate (10, 15, 30 Hz), add a fixed latency to camera tracks (0 to 150 ms) and simulate radar track dropouts lasting one to two seconds. Make sure the fuser predicts all tracks to a common fusion time before fusing them. Measure fused error and track continuity in each case. This shows how robust the architecture is to realistic network and sensor behaviour.

   Checkpoint: Plots of fused error against latency and against dropout length for the fusion rules.

8. 8

   Evaluate overall track quality

   Compute GOSPA with trackGOSPAMetric for local, fused and central tracks across all scenarios and at least 50 seeds. Also count track breaks and duplicate fused tracks, which appear when the fuser fails to associate two local tracks of the same vehicle. Summarise everything in one table with means and confidence intervals so the architectures can be compared on one page.

   Checkpoint: A summary table of GOSPA, RMSE, NEES, breaks and duplicates for all architectures and rules.

9. 9

   Analyse failure cases

   Replay runs with duplicate fused tracks or large errors on a bird's-eye plot. Typical causes include a large camera range error putting its track outside the fuser's gate, or latency causing misalignment during braking. Explain each case and test one fix, such as a larger gate or latency compensation, showing its effect on the metrics. Keep the fixes separate from the main results.

   Checkpoint: Annotated failure cases with one tested fix and its measured effect.

10. 10

    Report

    Present the architecture, local trackers, fusion rules, benchmark, robustness studies and failure analysis. Explain clearly why correlated errors matter and how covariance intersection protects consistency. Recommend a fusion rule for a supplier-module architecture and state its cost in accuracy relative to central fusion. Release scenarios, seeds and scripts so the results can be reproduced. Include the data-volume comparison as a table, because the bandwidth argument is often what decides the architecture in a real programme. Add a short glossary of the fusion terms for readers new to the topic.

    Checkpoint: A report with a supported recommendation and reproducible code and data.

Testing & validation

• Local tracker accuracy showing complementary sensor strengths.
• RMSE and NEES for naive, cross-covariance and covariance intersection fusion.
• Comparison with central detection-level fusion, including data volume.
• Fused error under varying rates, latency and dropouts.
• GOSPA, track breaks and duplicate fused tracks over 50 seeds.

Expected results

A track-to-track fusion architecture with a quantified comparison of fusion rules and of track-level against central fusion, showing the accuracy, consistency and bandwidth trade-offs.

Take it further

• Add a corner radar as a third source with a partly overlapping field of view.
• Model the object-list messages on CAN FD and measure the effect of message scheduling.
• Use JPDA local trackers in clutter and compare fused results.
• Fuse tracks with different motion models in the two local trackers.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.