MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models

A new benchmark called MET-Bench reveals that current vision-language models struggle to track entity states across text and images, with a performance gap driven by weak visual reasoning rather than perception failures [1]. The benchmark, introduced in a paper posted to the arXiv preprint repository on 15 Feb 2025, evaluates how well models integrate textual and image-based state updates across three domains [1]. The work was last revised on 12 Jun 2026 [1]. Vanya Cohen is listed as the corresponding author [1]. Entity state tracking requires maintaining coherent representations of objects over time, a capability central to world modeling [1]. Prior evaluations focused on text-only scenarios, leaving multimodal tracking largely unmeasured [1]. MET-Bench fills that gap by testing models on sequences that mix written descriptions with visual changes [1]. The authors report a significant gap between text-based and image-based tracking performance [1]. They attribute the shortfall primarily to deficits in visual reasoning, not to an inability to perceive objects in images [1]. Explicit text-based reasoning strategies lifted scores, but limitations persisted on long-horizon multimodal tasks [1]. In a further experiment, the team applied reinforcement learning to open-source vision-language models [1]. The approach produced substantial gains within a single modality, yet the improvements did not transfer reliably when the input modality switched [1]. The findings point to a need for stronger multimodal representations and reasoning techniques [1]. The paper appeared on arXiv, an open-access repository that hosts preprints across physics, computer science, and related fields [5]. As of November 2024, the platform was receiving roughly 24,000 submissions per month [5]. arXiv papers are moderated but not peer-reviewed [5]. The repository passed two million articles by the end of 2021 [5]. MET-Bench’s first submission weighed 9,310 KB; the second and third versions were 599 KB and 240 KB, respectively [1]. The shrinking file sizes across revisions reflect typical manuscript compression and refinement cycles on the platform [1].