The problem
3-D detection from lidar gives object positions, sizes and headings directly in metres, which planning and collision functions need. PointPillars is widely used because it converts point clouds into a pseudo-image that a 2-D network can process quickly. Understanding its design choices and their effect on accuracy, small-object performance and latency is valuable for anyone working on lidar perception.
Objectives
- Set up KITTI data with train and validation splits and ground-truth database augmentation.
- Train PointPillars and reproduce a credible baseline.
- Run ablations of pillar size, point features and augmentation.
- Evaluate 3-D and BEV AP by class and difficulty, and errors by range.
- Profile latency and compare with a classical clustering baseline.
What you need
Tools and toolboxes
- Python — Data preparation, training scripts and analysis.
- PyTorch — PointPillars model implementation and training.
- Open3D — Visualising point clouds with predicted and ground-truth boxes.
- ONNX — Exporting the backbone for latency comparison.
Software
- KITTI 3-D object detection dataset
- Python 3 with PyTorch and an open-source 3-D detection codebase or a clean reimplementation
- NumPy, Numba or CUDA ops for voxelisation, Matplotlib
Extra parts and data
- A GPU with at least 8 GB memory — Training a 3-D detector on KITTI takes many hours even on a GPU.
Key concepts
- Pillars
- Vertical columns on a ground-plane grid. Points inside each pillar are encoded by a small PointNet into a feature vector, producing a 2-D pseudo-image.
- Anchor-based detection head
- Predefined boxes of typical sizes and orientations at each location; the network predicts class scores and offsets from these anchors.
- 3-D AP and BEV AP
- Average precision computed with 3-D IoU or bird's-eye IoU between boxes. KITTI uses IoU 0.7 for cars and 0.5 for pedestrians and cyclists.
- Ground-truth database sampling
- An augmentation that pastes object point clouds cut from other scans into the current scene, increasing the number of rare objects per batch.
Step-by-step guide
- 1
Plan the study
State the questions: can you reproduce a credible PointPillars baseline on KITTI, which design choices matter most for pedestrians and cyclists, and what is the latency on your GPU? Fix the evaluation protocol: the common train and validation split of the KITTI training set, official difficulty levels and AP with 40 recall positions. Plan ablations with the same training schedule for each.
Checkpoint: A study plan with questions, protocol and an ablation table to fill in.
- 2
Prepare the data
Organise KITTI velodyne scans, labels and calibration and create the train and validation split files. Convert labels into lidar coordinates with heading angles and filter points to the camera field of view, as the evaluation does. Build the ground-truth database by cutting points inside each labelled box for sampling. Visualise several scans with boxes in Open3D to confirm the transforms.
Checkpoint: Correctly placed boxes in visualised scans and a built ground-truth database.
- 3
Implement or configure the model
Set the point cloud range, for example 0 to 69 m forward and ±40 m sideways, and a pillar size of 0.16 m. Each point gets features x, y, z, reflectance, offsets from the pillar mean and from the pillar centre. A simplified PointNet layer encodes each pillar, features are scattered to a pseudo-image, a 2-D backbone with three downsampling blocks and upsampling produces features, and an anchor head predicts boxes.
Checkpoint: A forward pass on one batch producing outputs of the expected shapes.
- 4
Train the baseline
Train with focal loss for classification, smooth L1 for box regression and a direction classifier for heading, using the one-cycle learning-rate policy and the Adam optimiser for around 80 to 160 epochs. Use global rotation, scaling and flipping plus ground-truth sampling. Log losses and evaluate on validation every few epochs. Save the configuration and best checkpoint. Compare your validation AP with published PointPillars figures for the same split and explain any gap before starting ablations.
Checkpoint: Training curves and validation AP for cars at a credible level for this method.
- 5
Evaluate in detail
Compute 3-D and BEV AP for car, pedestrian and cyclist at easy, moderate and hard difficulty. Beyond AP, compute recall by distance band and number of points per object, and the heading and position error for true positives. Visualise false negatives and false positives in Open3D to understand them, for example sparse distant pedestrians or poles detected as pedestrians. Save the evaluation outputs so later ablations can be compared object by object, not only through summary AP.
Checkpoint: AP tables, recall by distance and point count and a categorised error gallery.
- 6
Ablate pillar size
Train with pillar sizes such as 0.12, 0.16 and 0.24 m, keeping everything else fixed and adjusting the grid so the range is unchanged. Compare AP per class and inference latency. Smaller pillars should help small objects at the cost of speed and memory. Plot AP against latency to show the trade-off. Also record GPU memory use during training and inference for each size, because memory often limits what an embedded target can run. Explain why the smallest pillars may not give the best result.
Checkpoint: An ablation table and plot of AP against latency by pillar size.
- 7
Ablate features and augmentation
Train variants without reflectance, without the pillar-centre offsets and without ground-truth sampling. Compare AP per class with the baseline. Where training time allows, repeat the most important variant with a second seed to check whether differences exceed run-to-run variation. Summarise which components matter most for pedestrians and cyclists. Keep the schedule, seed and evaluation script identical to the baseline for every variant. Present the results as AP differences from the baseline with the seed spread alongside, so the reader can see which changes are clearly larger than noise.
Checkpoint: A table of AP changes for each removed component, with seed variation where measured.
- 8
Profile latency
Time voxelisation, pillar encoding, backbone, head and post-processing (including NMS) separately on the GPU, averaging over the validation set after warm-up. Export the backbone and head to ONNX and compare inference time in ONNX Runtime. Report the total time per scan and whether it fits the typical 10 Hz lidar rate. Compare with the classical clustering pipeline's runtime and accuracy on the same scans.
Checkpoint: A per-stage latency breakdown and a comparison with the clustering baseline.
- 9
Test generalisation
Run the trained model on a different lidar dataset subset, such as nuScenes, after converting coordinates and restricting to the overlapping classes and field of view. Measure AP or recall and describe the drop due to different sensor height, beam count and point density. This shows the sensitivity of lidar detectors to the sensor setup. Visualise several nuScenes scans with predictions to show qualitatively how the different point pattern of the other sensor affects detection.
Checkpoint: Cross-dataset recall with a written explanation of the main causes of the drop.
- 10
Write the thesis chapter
Present the method, data preparation, baseline results, ablations, latency profile and cross-dataset test. Compare your numbers with published PointPillars results and explain any differences honestly. Release configurations and scripts. Discuss what would be needed for deployment, such as quantisation, TensorRT and a tracker downstream. State plainly where results fell short of published figures and what you believe caused it. Include the commands used to train and evaluate each configuration, and describe the hardware and software versions, so that another student can regenerate every table in the chapter from your released files.
Checkpoint: A chapter with reproducible configurations and complete result tables.
Testing & validation
- 3-D and BEV AP by class and difficulty on the KITTI validation split.
- Recall by distance and by point count per object.
- Ablation results for pillar size, features and augmentation.
- Per-stage latency and cross-dataset recall.
Expected results
A trained PointPillars detector with a credible KITTI baseline, ablations explaining which choices drive small-object accuracy and speed, a latency profile and a cross-dataset generalisation test.
Research angle for your M.Tech thesis
Research questions: which encoding and augmentation choices most affect small-object 3-D detection in pillar-based detectors, and how do they trade against latency and cross-sensor generalisation? Metrics: 3-D and BEV AP by class and difficulty, recall by range and point count, latency per stage. Baseline: a reproduced PointPillars configuration. Carefully controlled ablations with seed variation and a cross-dataset test, released with configurations, make a sound conference contribution.
Take it further
- Fuse camera image features into pillars and compare.
- Deploy on a Jetson board with TensorRT.
- Add a 3-D multi-object tracker and measure tracking metrics.
- Train on nuScenes lidar and compare class performance with KITTI.
Viva and review questions
Why does PointPillars convert the cloud to a pseudo-image?
So that fast, mature 2-D convolution backbones can be used instead of slower 3-D convolutions, which is the main reason for its speed.
Why is pedestrian AP lower than car AP?
Pedestrians are small and have few lidar points, especially at range, and are easily confused with poles and other thin objects.
What does ground-truth sampling do?
It pastes extra labelled objects from other scans into training scenes, increasing examples of rare classes and improving their recall.
Why might the model degrade on another dataset?
Different lidar height, beam count and point density change how objects look in the point cloud, which the model has not seen.
Learn it at DIYguru
These DIYguru courses and programmes teach what this project needs.