Cooperative Perception with V2X-Shared Object Lists

The problem

A car's sensors cannot see around a bus, a parked truck or a blind junction corner, all common in dense Indian traffic. Cooperative perception lets vehicles and roadside units share what they see, for example in a collective perception message, so the ego vehicle learns about hidden pedestrians or two-wheelers. In practice, shared data arrives late, some packets are lost and each sender's position is uncertain. How much benefit survives these effects is an open question.

Objectives

• Build CARLA scenarios with occlusions where sharing should matter.
• Run per-agent perception that produces object lists with uncertainties.
• Model V2X transmission with message rates, latency, packet loss and limited message size.
• Fuse received object lists with the ego vehicle's own tracks, compensating for latency and pose error.
• Measure occluded-object detection, warning lead time and track accuracy against an ego-only baseline.

What you need

Key concepts

Collective perception

Vehicles and roadside units broadcast lists of objects they detect, with positions, velocities and uncertainties, so receivers can extend their own view. ETSI defines a Collective Perception Message for this purpose.

Late fusion of object lists

Shared data is fused at the object level rather than raw sensor level, keeping messages small enough for the V2X channel.

Latency compensation

A received object's state is predicted forward from its timestamp to the ego vehicle's current time before fusion, using its velocity and uncertainty.

Pose uncertainty

A sender's own localisation error shifts every object it reports. Ignoring it causes duplicate or misplaced tracks in the receiver's map.

Step-by-step guide

1. 1

   Define research questions and scenarios

   State the questions: how much does cooperative perception improve detection of occluded vulnerable road users, and how does the gain degrade with latency, packet loss, penetration rate and pose error? Design CARLA scenarios: a pedestrian stepping out from behind a parked bus, a two-wheeler emerging at a blind junction, and a stalled vehicle beyond a curve. For each, define the hazard moment and the ground-truth visibility from each agent.

   Checkpoint: A scenario specification with hazard timing and which agents can see each hidden object.

2. 2

   Set up CARLA in synchronous mode

   Run CARLA with synchronous mode and a fixed time step (for example 0.05 s) so all agents advance together and results are repeatable. Spawn the ego vehicle, two or more connected vehicles and a roadside unit at the junction, each with a camera and a lidar. Use the Traffic Manager for background traffic with fixed seeds. Log ground truth positions of all actors every step.

   Checkpoint: A scripted run that reproduces the same scenario exactly on repeated executions.

3. 3

   Build per-agent perception

   Give each agent a detector producing an object list in its own frame: either a trained YOLO model on camera images with depth from lidar, or ground-truth objects filtered by actual sensor visibility with added noise. The second option gives controlled experiments; the first adds realism. Attach a covariance to each object from the detector's measured error. Each agent runs its own local tracker.

   Checkpoint: Per-agent object lists with measured detection accuracy against ground truth.

4. 4

   Model localisation error

   Add realistic errors to each agent's own pose estimate, such as a random position offset of up to a metre or two and a small heading error, slowly varying over time. Transform each agent's objects into a global frame using its estimated, not true, pose. This lets you study how pose error spreads into the shared objects, which is a major practical problem for cooperative perception.

   Checkpoint: Plots showing shared objects displaced in the global frame by the simulated pose error.

5. 5

   Build the V2X channel model

   Encode each agent's object list into a compact message with timestamp, sender pose, and object positions, velocities, classes and covariances. Simulate transmission at a fixed rate (for example 10 Hz) with configurable latency, random or bursty packet loss and a maximum message size that forces object prioritisation. Optionally replace this model with an OMNeT++ simulation for channel-load effects. Record every message sent and received with timestamps, so delivery can be analysed afterwards.

   Checkpoint: A channel model whose measured latency and loss match the configured values in logs.

6. 6

   Fuse received objects with ego tracks

   On the ego vehicle, predict each received object to the current time using its velocity, inflating its covariance with the elapsed time and sender pose uncertainty. Associate received objects with ego tracks by a gated global nearest neighbour step, updating matched tracks and starting new tracks for objects the ego cannot see. Use covariance intersection when the same object may be reported through several senders.

   Checkpoint: In the bus scenario, the hidden pedestrian appears in the ego's track list before it becomes visible to ego sensors.

7. 7

   Add a downstream warning

   Implement a simple collision warning on the fused track list based on predicted conflict and TTC with the ego path. Measure warning lead time for each hidden hazard with and without cooperative perception. This connects perception gains to a safety-relevant outcome and is clearer to examiners than detection metrics alone. Keep the warning thresholds identical in both modes so that only the perception input differs between them. Note any extra false warnings that shared objects introduce.

   Checkpoint: Warning lead times for each hazard in ego-only and cooperative modes.

8. 8

   Run the parameter study

   Sweep latency (0 to 300 ms), packet loss (0 to 50 per cent), penetration rate of connected vehicles (0 to 100 per cent) and pose error levels. For every combination, run at least 20 seeds per scenario. Record detection rate and time-to-first-detection for occluded objects, GOSPA of the fused track list, duplicate tracks and warning lead time. Vary one factor at a time around a nominal setting first, then run a reduced full-factorial set for interactions.

   Checkpoint: A complete results table for the parameter sweep with no missing combinations.

9. 9

   Study robustness measures

   Compare three fusion variants: naive fusion ignoring latency and pose error, latency-compensated fusion, and fully uncertainty-aware fusion with pose covariance and covariance intersection. Also test a simple misbehaviour case where one sender reports a fake object, and check whether a plausibility check against ego sensors catches it. Report each variant's effect on duplicates and false warnings. Use the same seeds and channel conditions for all three variants.

   Checkpoint: A comparison of the three fusion variants and the result of the plausibility check.

10. 10

    Analyse and report

    Plot benefit curves: warning lead time and occluded-object detection rate against latency, loss and penetration rate. Identify the operating region where cooperative perception is clearly helpful. Discuss limitations: simulated sensors and channel, CARLA town layouts that differ from Indian streets, and no real message standard implementation. Release scenarios, channel model and fusion code. Present a short section on what a real deployment would need, such as message standards, security credentials and roadside infrastructure, without claiming it has been tested. Keep the discussion tied to your measured curves.

    Checkpoint: Thesis chapters with benefit curves, limitations and released code.

Testing & validation

• Repeatable synchronous CARLA scenarios with logged ground truth.
• Occluded-object detection rate and time-to-first-detection, ego-only against cooperative.
• Warning lead time for each hidden hazard.
• Sensitivity to latency, packet loss, penetration rate and pose error.
• Duplicate tracks and false warnings for naive and uncertainty-aware fusion.

Expected results

A cooperative perception pipeline in CARLA with measured gains for occluded hazards and benefit curves showing how latency, packet loss, penetration and pose error limit those gains.

Take it further

• Share intermediate BEV features instead of object lists and compare bandwidth and accuracy.
• Replace the simple channel model with OMNeT++ for congested multi-vehicle channels.
• Add security checks for message authenticity and plausibility in line with ISO/SAE 21434 thinking.
• Build an Indian-style junction in RoadRunner and import it into CARLA.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.