From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

A research team proposes converting egocentric video into Temporal Action Graphs to improve action recognition, finding that vision-language models reason more effectively over symbolic structures than raw pixels [1]. The method, detailed in a paper submitted to arXiv on June 13, 2026, decouples visual perception from symbolic reasoning. A multi-stage prompting pipeline first generates dense natural language narratives over short temporal windows, creating a semantic bottleneck. Those narratives are then formalized into structured, open-vocabulary graph representations [1][2]. The approach was tested on the EGTEA and Epic-Kitchens-100 datasets, both standard benchmarks in the computer vision subdiscipline of activity recognition [1][3][4]. When provided with few-shot graph demonstrations, the symbolic representation unlocked efficient in-context learning, yielding what the authors describe as substantial accuracy gains over both zero-shot frame-based and graph-based inference [1][2]. In the zero-shot setting, graph-based reasoning remained competitive with pixel-based inference, even though the latter may have benefited from pretraining contamination [1]. The study evaluated 11 open-weight vision-language models drawn from 6 model families, with parameter counts ranging from 2B to 235B [1][2]. Large language models of this kind are trained with self-supervised learning on vast text corpora [10]. Across all tested configurations, the researchers concluded that current VLMs are more effective as symbolic reasoners than as direct visual observers [1][2]. By projecting video into the language domain, the technique offers a scalable, fine-tuning-free alternative to end-to-end approaches that the authors say better leverages these models' latent reasoning strengths [1][2]. The paper was posted through arXiv, the open-access e-print repository that has hosted over two million articles since its founding in 1991 and currently receives about 24,000 submissions per month [8]. The researchers, including corresponding author Bessie Dominguez-Dager, indicated that the code will be made public [1][2].