4-D Imaging Radar Point Clouds for Object Classification

The problem

4-D imaging radars produce sparse point clouds with elevation and Doppler, working in rain, fog and darkness where cameras and lidar struggle, but their points are far sparser and noisier than lidar. It is not yet clear how best to learn from them, how much each attribute contributes and how performance compares with lidar on the same scenes.

Objectives

• Prepare 4-D radar point cloud data with object labels.
• Train point-based and pillar-based networks for classification and detection.
• Measure the contribution of Doppler, RCS and multi-frame accumulation.
• Compare with a lidar detector on matched scenes.
• Evaluate robustness and runtime.

What you need

Key concepts

4-D imaging radar

A radar with enough virtual channels to estimate elevation as well as azimuth, producing points with range, azimuth, elevation and Doppler, plus RCS.

Doppler as a feature

Radial velocity per point helps separate moving objects from static background and can distinguish pedestrians, whose limbs give varied Doppler, from rigid objects.

Multi-frame accumulation

Merging points from several past frames, after ego-motion compensation, to densify the sparse cloud; moving objects smear unless their motion is compensated too.

Point-based versus pillar-based

Point-based networks such as PointNet++ process raw points; pillar-based networks grid them into a pseudo-image for 2-D convolution. They trade detail against speed.

Step-by-step guide

1. 1

   Define the questions and protocol

   State the questions: which network design best suits sparse 4-D radar, how much do Doppler, RCS and accumulation contribute, and how far behind lidar is radar on matched scenes and under adverse conditions? Fix the dataset splits, classes (car, pedestrian, cyclist), metrics (classification accuracy, 3-D and BEV AP, recall by range) and the number of seeds per experiment. Define in advance what size of difference you will treat as meaningful, given the expected seed variation.

   Checkpoint: A protocol document with questions, splits, classes, metrics and seeds.

2. 2

   Prepare and inspect the data

   Load the 4-D radar scans and labels, transform them into a common vehicle frame and inspect point counts per object by class and range. Plot Doppler and RCS distributions per class. Note the sparsity, for example how many pedestrians have fewer than five points, because that sets an upper bound on detection. Keep the official or a scene-separated train, validation and test split.

   Checkpoint: Statistics of points per object, Doppler and RCS per class, and fixed splits.

3. 3

   Build a classification benchmark

   Before detection, crop radar points inside each labelled box (with a margin) and train classifiers to identify the class from the points alone: a PointNet++ style network and a simple handcrafted-feature baseline using extent, point count, Doppler spread and RCS statistics with a gradient-boosted tree. This isolates how much class information sparse radar points carry. Report accuracy by range band and point count, since classification from three points is a very different task from classification from thirty.

   Checkpoint: Classification accuracy and confusion for learned and handcrafted models.

4. 4

   Train detection networks

   Adapt a pillar-based detector to radar by using per-point features x, y, z, Doppler (ego-motion compensated) and RCS, with larger pillars to suit the sparsity, and train a point-based detector for comparison. Use identical training budgets and augmentation (rotation, scaling, flipping). Log AP per class on validation and keep the best checkpoints. Tune anchor sizes or centre-based heads for radar, since box sizes are the same as for lidar but point support is much weaker. Record training time and memory for both networks.

   Checkpoint: Trained detectors of both types with validation AP curves.

5. 5

   Ablate radar attributes

   Retrain the better detector with Doppler removed, RCS removed, elevation removed (projecting to 3-D radar) and all three removed, using three seeds each. Report the AP change for each class. This shows what 4-D imaging adds over conventional radar and which attributes matter for pedestrians and cyclists in particular. Pay particular attention to pedestrians and cyclists, where Doppler from moving limbs and wheels may carry most of the class information. Plot the results by range band as well, since attribute value may change with distance.

   Checkpoint: An ablation table of AP by class with seed spread for each removed attribute.

6. 6

   Study multi-frame accumulation

   Accumulate the past one to five frames after ego-motion compensation, adding a time-offset feature to each point. Compare AP and recall for moving and static objects. Then try compensating object motion using per-point Doppler to reduce smearing. Measure the latency cost of larger inputs and find the best accumulation length. Report results for fast-moving and slow-moving objects separately, since accumulation smears fast objects most and can lower their AP even as it helps static ones. Choose a final setting from the trade-off.

   Checkpoint: AP and latency against the number of accumulated frames, with and without motion compensation.

7. 7

   Compare with lidar on matched scenes

   Where the dataset includes lidar, train a PointPillars lidar detector on the same scenes and split. Compare AP by class and range and recall for objects with few radar points. Where possible, select scenes with rain or darkness and compare degradation for both sensors. Report the comparison honestly, including where radar is far behind. Use the same evaluation script and range limits for both sensors so that differences cannot come from evaluation settings.

   Checkpoint: A radar against lidar comparison by class, range and condition.

8. 8

   Use simulation for controlled studies

   In MATLAB with Radar Toolbox and the Driving Scenario Designer, simulate radar detections for scripted scenarios with exact ground truth, such as a pedestrian crossing between parked cars at varying distances. Use them to probe failure cases the real data has too few of. Treat simulated results as supporting evidence only and state the gap between simulated and real radar points.

   Checkpoint: Results on targeted simulated scenarios with a statement of their limits.

9. 9

   Evaluate robustness and runtime

   Test the best model with added point dropout and position noise to emulate weaker sensors and multipath, and on scenes outside the training distribution. Measure inference time per frame on the GPU, broken into pre-processing, network and post-processing, and compare with the radar's frame rate. Identify the configuration that best balances accuracy and runtime. Compare runtime with the lidar detector on the same hardware to show the cost difference between the two sensors' pipelines.

   Checkpoint: Robustness curves and a runtime breakdown with a recommended configuration.

10. 10

    Write the thesis chapter

    Present data analysis, classification benchmark, detection results, attribute ablations, accumulation study, radar–lidar comparison, simulation probes and robustness. Discuss what the results imply for radar-only and radar-camera fusion designs, and how 4-D radar fits SOTIF arguments for adverse weather. Release code and configurations, and describe dataset preparation steps without redistributing data. State plainly where 4-D radar was not good enough for a given class or range, as these negative results guide sensor choices and future research as much as the positive ones.

    Checkpoint: A complete chapter with released code and clear conclusions on each question.

Testing & validation

• Classification accuracy from radar points alone, learned and handcrafted.
• 3-D and BEV AP by class and range for point- and pillar-based detectors.
• Attribute and accumulation ablations with seed variation.
• Radar against lidar comparison and runtime per frame.

Expected results

A 4-D radar perception study with trained detectors, quantified contributions of Doppler, RCS, elevation and accumulation, a matched comparison with lidar and runtime figures.

Take it further

• Fuse 4-D radar with camera features and compare with each sensor alone.
• Add a tracker that uses Doppler for faster track initiation.
• Self-supervised pretraining on unlabelled radar scans.
• Deploy the best model on an embedded GPU with TensorRT.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.