Comparing Human Gaze and Vision-Language Model Attention in Safety-Relevant Environments

Large vision-language models can identify regions of a scene that correspond to human visual attention in safety-relevant environments without requiring eye-tracking training data, according to a study comparing model-generated saliency maps with human gaze patterns. The study, posted to arXiv on 13 June 2026, collected eye-tracking data from ten participants who viewed 33 scene images depicting environments with varying levels of potential risk using Pupil Invisible wearable glasses [1][2]. Gaze coordinates were mapped onto stimulus images to produce population-averaged human gaze heatmaps. Separately, GPT-4o was prompted through the OpenAI Vision API to generate spatial predictions of visual attention, which were converted into saliency maps for comparison [1][2]. Spatial alignment was evaluated using four metrics. The Pearson correlation coefficient reached 0.515, the Normalised Scanpath Saliency (NSS) score was 0.988, Kullback-Leibler (KL) divergence measured 1.766, and the Area Under the Receiver Operating Characteristic Curve using the Judd formulation (AUC-Judd) registered 0.806 [1][2]. All values exceeded the AUC-Judd chance baseline of 0.5 [1][2]. A cross-model comparison included Gemini Pro, Gemini Flash, and Claude. Gemini Pro demonstrated the strongest spatial localisation on three of the four metrics, while GPT-4o produced the closest distributional match to human attention as measured by KL divergence [1][2]. Eye tracking, the process of measuring the point of gaze or motion of an eye relative to the head, is used across psychology, psycholinguistics, marketing, and product design [4]. The technology has also been examined for assistive applications such as controlling wheelchairs and robotic arms, and as a tool for early detection of autism spectrum disorder [4]. The study's approach bypasses the need for such specialised hardware, instead relying on model-generated predictions. Attention itself is not a unitary phenomenon but encompasses multiple processes including selective attention, sustained attention, and orienting, supported by distributed neural networks in frontal, parietal, and subcortical regions [3]. The finding that vision-language models can approximate these patterns without explicit training on eye-tracking data suggests a scalable alternative for applications where collecting human gaze data is impractical. The research sits at the intersection of computer vision and human-computer interaction, a field that examines how people operate and engage with computer systems and designs interfaces between users and machines [5]. OpenAI, the developer of GPT-4o, is a San Francisco-based AI research organization whose release of ChatGPT in November 2022 helped catalyze the current AI boom [8]. The broader field of generative AI, which includes large language models and vision-language models, has seen rapid adoption since the 2020s across sectors including healthcare, finance, and product design [9]. AI safety researchers have expressed concern that safety measures are not keeping pace with the rapid development of AI capabilities [10]. The study's focus on safety-relevant environments — scenes containing potential risks — aligns with broader efforts to understand how AI systems perceive and respond to hazardous situations.