The problem
A detector that runs well on a desktop GPU may miss its deadline on an in-vehicle computer with limited power and memory. Quantisation and engine optimisation speed it up but can reduce accuracy, especially on small or rare objects. ADAS engineers must show that latency stays within budget without unacceptable accuracy loss.
Objectives
- Define a latency and accuracy budget for a forward perception function.
- Export a trained detector to ONNX and build TensorRT engines at several precisions.
- Calibrate INT8 quantisation with a representative dataset.
- Measure accuracy, latency distribution, throughput and power for each engine.
- Profile the end-to-end pipeline and identify bottlenecks.
What you need
Tools and toolboxes
- Python — Export, calibration, evaluation and profiling scripts.
- Ultralytics YOLO — The trained detector and its validation tooling.
- ONNX — Interchange format and graph inspection before engine building.
- TensorRT — Building and running FP32, FP16 and INT8 inference engines.
- OpenCV — Camera or video capture, pre-processing and drawing.
Software
- JetPack with TensorRT, CUDA and cuDNN
- Python 3 with Ultralytics, ONNX, ONNX Runtime and OpenCV
- trtexec and tegrastats for timing and power
- A labelled test set, for example from IDD or BDD100K
Extra parts and data
- NVIDIA Jetson development board (for example an Orin Nano class board) — The embedded GPU target on which latency and power are measured.
- USB camera or recorded dashcam video — Input for the end-to-end pipeline; recorded video makes runs repeatable.
Key concepts
- TensorRT engine
- A hardware-specific optimised inference plan built from a network graph, with fused layers and tuned kernels for one GPU and precision.
- INT8 calibration
- Running representative images through the network to choose scale factors that map floating-point activations to 8-bit integers with minimal error.
- Tail latency
- The 95th or 99th percentile of frame time. Safety functions must meet their deadline almost always, so the tail matters more than the average.
- End-to-end latency
- Time from image capture to usable output, including decoding, pre-processing, inference, post-processing and NMS, not inference alone.
Step-by-step guide
- 1
Set the budget and test protocol
Define the function, for example forward vehicle and pedestrian detection feeding a collision warning, with a budget such as 33 ms end-to-end at the 99th percentile and a maximum mAP drop of one or two points from the FP32 reference. Fix the board power mode, input resolution and test set for all experiments. Write down how latency is measured: wall-clock timestamps at capture and at output, over at least 1,000 frames after warm-up.
Checkpoint: A protocol document with budgets, power mode, resolution and measurement method.
- 2
Prepare the model and reference accuracy
Start from a detector fine-tuned on driving data, such as a YOLO model trained on IDD or BDD100K. Measure its [email protected] and [email protected]:0.95 and per-class recall on the test set in PyTorch on a desktop GPU; this is the reference. Keep a fixed calibration subset of a few hundred training images, separate from the test set, for INT8 calibration. Record the exact pre-processing (resize method, letterboxing, normalisation), because any mismatch on the board will appear as accuracy loss wrongly blamed on quantisation.
Checkpoint: Reference metrics on the test set and a fixed calibration image list.
- 3
Export to ONNX and verify
Export the model to ONNX with a fixed input shape and a supported opset. Inspect the graph for unsupported operators and decide whether NMS stays in the graph or runs on the CPU. Run the ONNX model in ONNX Runtime on a sample of images and compare boxes and scores with PyTorch outputs to confirm the export is numerically faithful before moving to the board.
Checkpoint: ONNX outputs matching PyTorch within a small tolerance on sample images.
- 4
Build FP32 and FP16 engines
On the Jetson board, build TensorRT engines from the ONNX file at FP32 and FP16 using trtexec or the TensorRT Python API, with a workspace size suited to the board memory. Record build settings in a configuration file. Use trtexec to report raw inference latency and throughput for each engine, then run the test set through each to measure mAP, so accuracy and speed are tied to the same engine file.
Checkpoint: Two engines with measured inference latency and test-set mAP.
- 5
Calibrate and build the INT8 engine
Implement an INT8 calibrator that feeds the calibration images with the same pre-processing as inference and caches the scale factors. Build an INT8 engine and measure mAP and per-class recall. If accuracy drops beyond budget, try a different calibration method, a larger calibration set or leaving sensitive layers such as the detection head in FP16 with mixed precision, and record each attempt.
Checkpoint: An INT8 engine with measured accuracy and a log of calibration choices tried.
- 6
Build the end-to-end pipeline
Write a pipeline that captures frames (camera or video file), pre-processes on the GPU where possible, runs the engine, decodes outputs, applies NMS and draws or publishes results. Timestamp each stage. Run capture and inference in separate threads with a small queue so a slow stage is visible rather than hidden. Ensure the pipeline drops stale frames rather than building an ever-growing backlog.
Checkpoint: A running pipeline with per-stage timestamps logged for every frame.
- 7
Measure latency, throughput and power
For each precision, run the pipeline for at least 1,000 frames and compute mean, median, 95th and 99th percentile latency and frames per second. Record GPU and CPU load and board power with tegrastats during the runs. Repeat with a warm board after ten minutes to see thermal effects. Present results in one table so accuracy, latency and power can be compared directly.
Checkpoint: A table of accuracy, latency percentiles, throughput and power for FP32, FP16 and INT8.
- 8
Profile and remove bottlenecks
Use the stage timestamps to find where time goes: often pre-processing on the CPU, NMS or memory copies rather than inference. Try targeted fixes such as GPU resizing, pinned memory, a reduced input size or batched NMS, measuring each change in isolation. Stop when the 99th percentile meets the budget or when the remaining cost is clearly in inference itself.
Checkpoint: A before-and-after profile showing the effect of each optimisation.
- 9
Check accuracy where it matters
Beyond overall mAP, compare per-class recall and recall on small objects between precisions, because quantisation errors often hit small and distant objects first. Inspect cases where FP32 detects a pedestrian and INT8 does not. Decide which precision is acceptable for the safety function and justify the choice with both the latency and the safety-relevant accuracy numbers. Include a short table of the specific frames where precisions disagree, with images, so the reader can judge their severity.
Checkpoint: Per-class and small-object recall by precision with a justified final choice.
- 10
Write the thesis chapter
Document the budget, export and engine build process, calibration experiments, accuracy and latency results, profiling and the final configuration. Include the scripts and engine build settings so the work can be reproduced on the same board. State the limitations: one model family, one board and recorded rather than live camera input. Discuss what else a production system would need, such as watchdogs and deterministic scheduling.
Checkpoint: A chapter with complete tables, build configurations and released scripts.
Testing & validation
- mAP and per-class recall for FP32, FP16 and INT8 against the PyTorch reference.
- Latency percentiles (mean, 95th, 99th) and throughput, end to end.
- Power draw and thermal behaviour during sustained runs.
- Per-stage profile before and after optimisation.
Expected results
A detector deployed on an embedded GPU with measured accuracy, tail latency, throughput and power at three precisions, a profiled end-to-end pipeline and a justified precision choice against an ADAS budget.
Research angle for your M.Tech thesis
Research questions: which layers and object categories are most sensitive to INT8 quantisation in driving detectors, and can mixed precision recover safety-relevant recall while meeting a tail-latency budget? Metrics: mAP, small-object and pedestrian recall, 99th percentile latency, power. Baselines: FP32 and FP16 engines on the same board. A layer-sensitivity study with a reproducible mixed-precision recipe, reported with tail latency rather than averages, is a useful contribution for an embedded vision venue.
Take it further
- Add a tracker and measure its cost within the budget.
- Compare with a second model family under the same protocol.
- Run two networks concurrently (detection and segmentation) and study contention.
- Use quantisation-aware training and compare with post-training INT8.
Viva and review questions
Why report 99th percentile latency rather than the mean?
A safety function must meet its deadline almost every frame; the mean can look fine while occasional slow frames break the budget.
What does INT8 calibration do?
It runs representative images through the network to choose scale factors that map floating-point activations into 8-bit integers with as little error as possible.
Why can quantisation hurt small objects first?
Small objects produce weaker, more subtle activations, and the reduced precision can push their scores below the detection threshold.
Why is the TensorRT engine built on the board itself?
Engines are optimised for a specific GPU architecture and software version, so an engine built elsewhere may not run or may not be optimal.
Learn it at DIYguru
These DIYguru courses and programmes teach what this project needs.