Lidar–Camera Extrinsic Calibration and Projection

The problem

Camera–lidar fusion only works if each lidar point can be placed accurately in the image. A small rotation error shifts points by many pixels at range, so fused objects get wrong depth or colour. Calibration must be accurate, repeatable and checkable, and engineers need to know how much error downstream fusion can tolerate.

Objectives

• Collect synchronised lidar and camera data of a checkerboard target.
• Estimate intrinsics and the lidar-to-camera transform with Lidar Toolbox.
• Validate with projection, reprojection error and edge alignment on held-out data.
• Compare with a targetless edge-based refinement.
• Quantify the effect of calibration error on fused object depth.

What you need

Key concepts

Extrinsic transform

The rotation and translation that map points from the lidar frame into the camera frame, often written as a 4×4 homogeneous matrix.

Plane and edge correspondence

The board's plane and edges are found in the lidar cloud and in the image; aligning them constrains the transform.

Reprojection error

The pixel distance between projected lidar features and their image counterparts, a direct measure of calibration quality.

Targetless calibration

Refining the transform using natural features such as depth discontinuities and image edges, useful for checking calibration in the field.

Step-by-step guide

1. 1

   Define requirements and data plan

   Set accuracy targets, for example mean reprojection error of a few pixels and rotation error below a few tenths of a degree, and explain them in terms of lateral error at 30 m. Plan data collection: 20 to 40 board poses at different distances, positions and tilts covering the overlapping field of view, with 25 per cent of poses held out for validation. Note how sensors are time-synchronised.

   Checkpoint: A requirements table and a pose plan with held-out poses identified.

2. 2

   Collect or select synchronised data

   Record the board in each pose, keeping it still while both sensors capture, or select KITTI calibration frames with the board visible. Pair each image with the lidar scan closest in time and discard pairs with motion or partial visibility. Store pairs with indices and record board dimensions and square size exactly, since they enter the estimation. Measure or estimate the time offset between sensors, because even a few tens of milliseconds causes visible misalignment if the rig moves.

   Checkpoint: A clean set of synchronised image–scan pairs with board dimensions recorded.

3. 3

   Calibrate camera intrinsics

   Use the Camera Calibrator app or estimateCameraParameters with the checkerboard images to get the camera matrix and distortion coefficients, rejecting images with high reprojection error. Undistort images before the extrinsic step. If using KITTI, compare your intrinsics with the provided ones to sanity-check the process. Capture the board across the whole image, including corners where distortion is strongest, and keep the focus fixed throughout. Report the mean reprojection error and the uncertainty of each intrinsic parameter, since intrinsic errors feed directly into the extrinsic estimate.

   Checkpoint: Intrinsics with low reprojection error, compared with any provided reference.

4. 4

   Detect the board in both sensors

   Detect checkerboard corners in images and compute the 3-D board corners in the camera frame from the known board size. In each lidar scan, crop a region around the board, fit its plane with RANSAC and extract its edges and corners, using detectRectangularPlanePoints in Lidar Toolbox. Visually check a sample of detections in both sensors and remove failures. Record the number of lidar points on the board for each pose, as poses with very few points weaken the estimate.

   Checkpoint: Board detections in images and point clouds checked visually for each retained pose.

5. 5

   Estimate the extrinsic transform

   Run estimateLidarCameraTransform with the image and lidar board features from the calibration poses, or use the Lidar Camera Calibrator app, and save the transform. Report the residual errors the tool provides: translation, rotation and reprojection. Repeat with different subsets of poses to see how stable the estimate is and how many poses are needed. Plot the estimated rotation and translation against the number of poses used, which shows when adding poses stops improving the result and how many the procedure really needs.

   Checkpoint: An estimated transform with residual errors and a stability analysis across pose subsets.

6. 6

   Validate on held-out poses

   Project lidar points of held-out poses into the images with projectLidarPointsOnImage, colour them by depth and overlay. Compute reprojection error of projected board corners against detected image corners. Also project points from full scenes and check that depth edges at poles, vehicles and building corners line up with image edges, which tests calibration beyond the board. Report errors separately for near and far poses, because rotation errors become more visible at longer range.

   Checkpoint: Held-out reprojection error and overlay images showing aligned depth edges.

7. 7

   Implement targetless refinement

   Implement an edge-alignment cost: extract depth discontinuities in the lidar scan and compute an image edge map with distance transform. Optimise the six transform parameters to maximise alignment over several natural scenes, starting from the target-based result and from a deliberately perturbed one. Compare the refined result with the target-based transform on held-out reprojection error. Report how many natural scenes are needed for the optimisation to converge reliably and how sensitive it is to scene content.

   Checkpoint: Refinement results from both starting points with held-out errors compared.

8. 8

   Quantify downstream impact

   Perturb the transform by small rotations (0.2 to 1 degree) and translations (1 to 5 cm) and measure the effect on a fused task: assign lidar depth to camera detection boxes on KITTI frames and compute depth error against labels. Plot depth error and assignment failures against perturbation size. This gives a tolerance specification for calibration based on fusion performance.

   Checkpoint: Plots of fused depth error against calibration perturbation with a derived tolerance.

9. 9

   Check repeatability

   Repeat the full calibration on a second session, or with a different subset of KITTI drives, and compare the transforms. If using a lab rig, remount the camera and repeat. Report the difference in rotation and translation and whether it is within the tolerance from the previous step. Recommend how often calibration should be checked and what quick check can be done in the field.

   Checkpoint: A repeatability table and a recommended field check procedure.

10. 10

    Write the thesis chapter

    Present the calibration model, data collection, intrinsic and extrinsic estimation, validation, targetless refinement, downstream tolerance study and repeatability. Include overlays, error tables and scripts. State the limitations: a single sensor pair, static poses and synchronisation assumptions. Make the tolerance specification and field check the practical outcomes. Include a one-page calibration procedure with the number of poses, board size, acceptance thresholds and the field check, so another team can apply it to a new sensor pair. Compare the final errors with the requirements from the first step and explain any shortfall.

    Checkpoint: A chapter with a tolerance specification and reproducible scripts.

Testing & validation

• Reprojection error on held-out board poses.
• Edge alignment on natural scenes.
• Target-based against targetless refinement accuracy.
• Fused depth error against calibration perturbation, and repeatability across sessions.

Expected results

A validated lidar–camera transform with held-out error measures, a targetless refinement method, a tolerance specification derived from fusion performance and a repeatability check.

Take it further

• Add time-offset estimation between the sensors.
• Calibrate a radar to the same frame using corner reflectors.
• Monitor calibration online and trigger recalibration.
• Use the calibration for a camera–lidar fused detector.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.