Pedestrian and Vehicle Detection with YOLO on Indian Road Video

The problem

Forward collision warning and emergency braking need reliable detection of road users. Indian traffic contains classes and densities that general-purpose detectors trained on other countries handle poorly, such as auto-rickshaws, dense two-wheeler traffic and pedestrians close to moving vehicles. Measuring and improving detection on local data is a core perception skill.

Objectives

• Prepare an IDD subset in YOLO format with a defined class list.
• Establish a baseline with a pretrained YOLO model before fine-tuning.
• Fine-tune and evaluate with [email protected], [email protected]:0.95 and per-class recall.
• Analyse errors by object size, occlusion and lighting.
• Measure frame rate and test on unseen recorded video.

What you need

Key concepts

Single-stage detector

YOLO predicts boxes and class scores in one pass over the image, which makes it fast enough for real-time video.

IoU

Intersection over union of a predicted and a true box. A detection counts as correct when IoU exceeds a threshold such as 0.5.

mAP

Mean of the average precision across classes, where average precision summarises the precision–recall curve. [email protected]:0.95 averages over several IoU thresholds and rewards tighter boxes.

Non-maximum suppression

Removes overlapping duplicate boxes for the same object, keeping the highest-scoring one.

Step-by-step guide

1. 1

   Define classes and requirements

   Choose the class list from IDD that matters for a forward collision function: person, rider, motorcycle, bicycle, autorickshaw, car, truck and bus, merging rare classes where needed. Set requirements in words and numbers, for example a minimum recall for pedestrians at a given confidence threshold and at least 20 frames per second on the target GPU. Note that missed pedestrians are worse than false alarms for this use, which affects threshold choice later.

   Checkpoint: A class list, the merge rules and written recall and speed targets.

2. 2

   Convert IDD to YOLO format

   Parse the IDD detection annotations and convert each box to YOLO format: class index followed by normalised centre x, centre y, width and height. Keep the dataset's official train and validation split and hold back part of validation as a final test set if no labelled test split is available. Write a dataset YAML with paths and class names, and plot class counts and box-size distributions to see imbalance and the share of small objects.

   Checkpoint: A dataset YAML that Ultralytics loads without errors, plus class and size histograms.

3. 3

   Measure a pretrained baseline

   Run a pretrained COCO YOLO model (for example a small 'n' or 's' variant) on the test set, mapping COCO classes to yours where possible (person, bicycle, motorcycle, car, bus, truck). Classes such as autorickshaw have no COCO equivalent and will show zero recall, which is part of the finding. Record [email protected] and per-class recall so fine-tuning has a clear baseline.

   Checkpoint: Baseline metrics per class, including the classes COCO cannot detect.

4. 4

   Fine-tune the detector

   Fine-tune from the pretrained weights with image size 640, a batch size the GPU allows and around 50 to 100 epochs with early stopping on validation mAP. Keep the default augmentations (mosaic, HSV jitter, scaling) and log loss curves. Run at least two model sizes so the accuracy against speed trade-off can be compared. Save the best weights and the exact training configuration for reproducibility.

   Checkpoint: Training curves and best checkpoints for two model sizes.

5. 5

   Evaluate on the test set

   Validate each fine-tuned model on the held-out test set and report precision, recall, [email protected] and [email protected]:0.95 per class. Plot precision–recall curves for person and rider. Choose an operating confidence threshold that meets the pedestrian recall target and report the resulting precision, because this is the number that matters for a warning function rather than mAP alone. Report the threshold value itself alongside the metrics, because a later deployment must use exactly the same value to reproduce these numbers.

   Checkpoint: Per-class metrics, PR curves and a justified operating threshold.

6. 6

   Analyse errors by size and condition

   Split ground-truth objects by box height (small, medium, large) and by occlusion where IDD provides it, and compute recall for each group. Inspect false negatives and false positives visually and categorise them: distant pedestrians, crowds merged into one box, riders confused with motorcycles, shadows or posters detected as people. Use the findings to decide whether a larger input size or more data for one class would help most.

   Checkpoint: Recall by object size and a counted list of error types with example images.

7. 7

   Test on unseen video

   Run the chosen model on recorded dashcam video from a different city or camera than IDD. Measure frames per second end to end, including decoding and drawing, with OpenCV timing. Label a few hundred frames sampled from the video for the person and vehicle classes and compute recall and precision there, so the generalisation gap from the dataset to new footage is measured rather than guessed.

   Checkpoint: Frame rate and recall on independently labelled frames from new video.

8. 8

   Export and compare inference speed

   Export the model to ONNX with the Ultralytics export function and compare inference time with PyTorch on the same hardware at image sizes 640 and 480. Check that detections from the exported model match the original on a sample of images. Report latency per frame and how the choice of model size and image size trades accuracy for speed, relating the result to the 20 frames per second target.

   Checkpoint: A table of latency and mAP for each model and image size, with matching exported outputs.

9. 9

   Report and demo

   Write up the class choice, conversion, baseline, fine-tuning, metrics, error analysis and unseen-video test. Include a short demo video with boxes, class labels and confidence. State the limitations, such as daylight-heavy data, a single camera height and annotation noise in crowded scenes. Explain how this detector would feed a tracker and a collision warning, and why detection alone is not enough to brake.

   Checkpoint: A report with reproducible training commands, metrics tables and a demo video.

Testing & validation

• [email protected] and [email protected]:0.95 per class on a held-out IDD test set.
• Pedestrian and rider recall at the chosen operating threshold.
• Recall by object size and occlusion.
• Frame rate and recall on independently labelled unseen video.

Expected results

A fine-tuned YOLO detector for Indian road users with measured gains over a COCO baseline, per-class and per-size metrics, an operating threshold chosen for safety, and a real-time video demo.

Take it further

• Add a multi-object tracker (for example ByteTrack) and measure ID switches.
• Deploy on a Jetson board with TensorRT and compare frame rate.
• Add night-time data and measure the change in recall.
• Estimate time to collision from box growth for a simple warning.

Viva and review questions

Learn it at DIYguru

These DIYguru courses and programmes teach what this project needs.