Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models

A new study posted to the arXiv preprint server demonstrates that vision language models can decode pedestrian crossing intentions from first-person video, achieving a 14.5% accuracy improvement over a specialized transformer baseline when guided by eye gaze and ego motion cues [1][2]. The work, submitted on June 8, 2026, frames pedestrian intent prediction as a closed-ended visual question answering task using short egocentric video clips [1][2]. Egocentric vision captures a first-person perspective of human perception and decision making, but its application to traffic-safety prediction has remained largely unexamined [2]. The researchers first benchmarked three families of state-of-the-art vision language models in a zero-shot setting, finding that they delivered only moderate gains over random guessing and showed limited higher-level traffic reasoning [2]. Motivated by those results, the team applied parameter-efficient fine-tuning to adapt the models to the target task. The fine-tuned models substantially outperformed their zero-shot counterparts and achieved a 9% accuracy improvement over a specialized transformer-based baseline [2]. Incorporating additional contextual cues — including ego motion, vehicle motion, and eye gaze — further lifted predictive performance [2]. The best result came from the fine-tuned Qwen3-VL-2B model, which, when guided by eye gaze and ego motion, reached a 14.5% accuracy improvement over the transformer baseline, establishing what the authors describe as a new state of the art for egocentric pedestrian intent decoding [2]. Vision language models are a class of models that combine visual understanding with natural language processing, building on the transformer architecture that underpins large language models [8]. Large language models are neural networks trained on vast text corpora and are often fine-tuned for specific tasks after pre-training [8]. The study’s use of parameter-efficient fine-tuning aligns with broader practices in the field, where fine-tuning adapts a general model to a narrow domain without retraining from scratch [8]. The paper appears on arXiv, an open-access repository that hosts electronic preprints across disciplines including computer science, physics, and mathematics [6]. As of late 2024, arXiv was receiving approximately 24,000 submissions per month and had surpassed two million total articles [6]. Papers on arXiv are moderated but not peer-reviewed before posting [6]. The repository also supports arXivLabs, a framework launched in 2020 that allows community collaborators to develop experimental tools — such as citation explorers and code finders — that appear on article record pages [5][4].