The problem
Most trajectory prediction research uses Western datasets with lane-disciplined cars. Indian traffic has dense, heterogeneous road users who weave, filter between lanes and share space with pedestrians. Predictors trained elsewhere may fail here, yet there is limited labelled trajectory data. This project builds data, a model and an evaluation specific to these conditions.
Objectives
- Assemble a trajectory dataset from recorded Indian traffic video with detection and tracking.
- Characterise behaviours that differ from Western datasets.
- Train a multi-agent, multi-modal predictor with agent-type awareness.
- Compare with constant-velocity, social and transferred baselines.
- Evaluate per agent type and per traffic density.
What you need
Tools and toolboxes
- Python — Data processing, model training and evaluation.
- PyTorch — Training the prediction network.
- Ultralytics YOLO — Detecting road users in video, fine-tuned on IDD classes such as auto-rickshaw and motorcycle.
- OpenCV — Homography estimation to convert image tracks to ground-plane coordinates.
- nuScenes (optional) — A Western reference dataset for transfer experiments.
Software
- Python 3 with PyTorch, OpenCV and NumPy
- Ultralytics YOLO with a multi-object tracker such as ByteTrack
- IDD (India Driving Dataset) for detector fine-tuning
Extra parts and data
- Recorded traffic video from fixed elevated viewpoints or drone footage obtained with permission — Trajectory data needs a stable bird's-eye-like view; recording must follow local regulations and privacy rules.
- GPU workstation — Detection, tracking and model training on large video sets need a GPU.
Key concepts
- minADE and minFDE
- The average and final displacement errors of the best of k predicted trajectories; standard metrics for multi-modal prediction.
- Heterogeneous agents
- Road users with very different sizes, speeds and manoeuvrability, which the model must distinguish to predict well.
- Social interaction modelling
- Encoding how nearby agents influence each other, for example with attention or graph neural networks.
- Domain shift
- Differences between training and target data distributions, which make transferred models perform worse.
Step-by-step guide
- 1
Define the problem and gap
Review trajectory prediction literature and existing datasets, and identify gaps for dense heterogeneous traffic. Define the task: predict 3 to 5 s of future positions from 2 s of history for every agent in a scene, at a set frame rate. Define agent types and evaluation splits by location and density. State hypotheses about which behaviours, such as filtering, matter most.
Checkpoint: A proposal chapter with task definition, hypotheses and planned splits.
- 2
Collect video and plan ethics
Record traffic at several junctions and road segments from elevated fixed viewpoints, or use drone footage only where permitted. Obtain permissions and follow privacy rules: avoid identifiable faces in outputs and store raw video securely. Measure ground reference points at each site for homography estimation. Document camera height, angle, time of day and conditions. Record at different times of day to capture varying density.
Checkpoint: Recorded video with site documentation, reference points and an approved data handling plan.
- 3
Detect and track road users
Fine-tune a YOLO detector on IDD and a few hundred annotated frames from your own video, with classes for car, bus, truck, motorcycle, bicycle, auto-rickshaw and pedestrian. Track with ByteTrack or a similar tracker. Evaluate detection mAP and tracking quality (MOTA, IDF1) on a hand-labelled subset. Fix systematic errors before generating trajectories, since prediction quality depends on them. Note which classes are most often confused.
Checkpoint: Detection mAP and tracking MOTA and IDF1 on a labelled subset.
- 4
Build the trajectory dataset
Convert tracks to ground-plane coordinates using homographies from the reference points, smooth them with a Kalman smoother, and resample to a fixed rate. Remove short or broken tracks. Split into scenes with history and future windows. Compute statistics: speed and acceleration distributions per agent type, density, lateral motion and interaction frequency, and compare them with a Western dataset. Keep raw and cleaned tracks for audit.
Checkpoint: A cleaned trajectory dataset with documented statistics and comparison plots.
- 5
Implement baselines
Implement constant-velocity and constant-acceleration baselines, a social pooling or attention-based model, and a model pre-trained on nuScenes and applied without fine-tuning. Evaluate all on your test split with minADE, minFDE and miss rate at k equals 1 and 6. This establishes how much domain shift matters. Use the same history and prediction lengths, coordinate frames and test scenes for every method, so the numbers are directly comparable with your proposed model.
Checkpoint: A baseline results table with transfer and in-domain numbers.
- 6
Design the proposed model
Design a model that encodes each agent's history with type embeddings and size, models interactions with a graph or transformer over nearby agents, and uses a scene context such as a rasterised drivable area if available. Decode multiple trajectories with probabilities. Train with a winner-takes-all loss plus classification of modes, and tune on validation data. Run ablations that remove type embeddings or interaction encoding.
Checkpoint: A trained model with validation metrics better than the baselines.
- 7
Evaluate thoroughly
Report metrics per agent type, per density band and per location, on held-out locations to test generalisation. Analyse calibration of mode probabilities. Visualise success and failure cases, especially two-wheelers filtering between vehicles and pedestrians crossing mid-block. Compare with fine-tuned transferred models to judge whether a dedicated model is needed. Report confidence intervals from bootstrapping over scenes, and check whether the improvement over baselines holds for every agent type.
Checkpoint: Breakdown tables, calibration plots and illustrated cases.
- 8
Test usefulness for planning
Feed predictions into a simple planner, such as a Frenet sampler with collision checking, in replayed scenes. Compare how often the planner would face predicted conflicts that did not occur, or miss conflicts that did, with each predictor. This links prediction metrics to downstream safety relevance. Keep the planner fixed and change only the predictor, and report results separately for scenes with vulnerable road users, since those conflicts matter most for safety.
Checkpoint: A planning-relevance analysis showing false and missed conflicts per predictor.
- 9
Write up and release
Write the thesis: data collection, tracking quality, dataset statistics, models, results and planning analysis. Release processed trajectories and code where permissions allow, with privacy protections. Discuss limitations such as viewpoint coverage, tracking errors and seasonal or regional differences. Include a datasheet describing how the data was collected, site conditions, tracking quality, known biases and intended use, so that others can judge whether it suits their work.
Checkpoint: Thesis chapters, a paper draft and a documented dataset release plan.
Testing & validation
- Detection mAP and tracking MOTA and IDF1 on labelled subsets.
- minADE, minFDE and miss rate per agent type, density and location.
- Calibration of predicted mode probabilities.
- Downstream false and missed conflicts in a planner.
Expected results
A trajectory dataset of Indian mixed traffic with documented quality, a heterogeneous-agent prediction model with evaluation by type and density, and evidence on how much domain shift affects transferred models.
Research angle for your PhD
Research questions: how much do predictors trained on lane-disciplined traffic degrade in dense heterogeneous traffic, and which modelling choices, such as type-aware embeddings and interaction encoders, recover performance? Metrics: minADE, minFDE, miss rate and calibration per agent type and density, plus downstream conflict errors. A new, carefully documented trajectory dataset with tracking-quality figures is itself a strong contribution, provided privacy and permission requirements are met.
Take it further
- Add ego-vehicle camera data to predict from the vehicle's view.
- Model lane-less behaviour with drivable-area context.
- Use the model as a reactive agent simulator for planning tests.
Viva and review questions
Why report metrics per agent type?
Pedestrians, two-wheelers and buses behave very differently; an overall average can hide poor performance on the most vulnerable or erratic users.
How do tracking errors affect your results?
ID switches and jitter corrupt both training labels and ground truth; you therefore measure tracking quality and smooth or remove poor tracks.
Why test on held-out locations?
Models can memorise site-specific paths; testing on new sites shows whether they generalise.
Learn it at DIYguru
These DIYguru courses and programmes teach what this project needs.