OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning

A new benchmark and data-generation pipeline called OmniTraffic aims to close the gap between how multimodal language models perceive traffic scenes and the spatio-temporal reasoning required for real-world driving decisions, according to a paper posted to the arXiv preprint repository [1]. The system is built around 12 real-world intersections reconstructed as editable 3D environments and supplemented with surveillance footage from two countries [1][2]. OmniTraffic defines a three-level task hierarchy: scene perception, multi-view and temporal reasoning, and decision support [1][2]. Using structured traffic metadata, the pipeline generates synchronized multi-view visual-question-answering samples that cover vehicle states, lane functions, view-to-bird’s-eye-view correspondence, temporal dynamics, and signal-phase analysis [1][2]. The result is a corpus of 8 million VQA samples and a 3,000-item human-verified test set [1][2]. Eleven frontier multimodal large language models were evaluated on the benchmark [1][2]. The authors report a large human–model gap, with the most pronounced failures occurring in topology-grounded and spatio-temporal reasoning tasks [1][2]. When a lightweight MLLM was fine-tuned on simulated OmniTraffic data, its performance on real-world traffic scenes improved, suggesting that simulation-generated supervision can strengthen traffic-specific multimodal reasoning [1][2]. The paper was submitted to arXiv’s Computer Vision and Pattern Recognition section on 14 June 2026 [1]. arXiv, which began operating in 1991, is an open-access repository of electronic preprints that are moderated but not peer-reviewed; it passed two million articles by the end of 2021 and currently receives about 24,000 submissions per month [6]. The OmniTraffic paper appears with the standard arXivLabs integration, a framework launched in 2020 that allows community collaborators to build experimental tools on top of the repository’s article pages while adhering to arXiv’s values of openness, community, excellence, and user-data privacy [4]. Beyond its fixed dataset, OmniTraffic is designed as an extensible pipeline. Users can configure intersections, camera views, traffic demands, signal phases, visual conditions, and rare events, making it possible to tailor evaluation scenarios to specific research questions [1][2].