SER: Learning to Ground Video Reasoning with Semantic Evidence Rewards

A new training method called Semantic Evidence Reward (SER) aims to improve how machine learning models reason about video content by evaluating the semantic quality of their evidence, rather than relying solely on geometric overlap metrics [1]. The approach, detailed in a paper submitted to the arXiv preprint server on 23 Jun 2026, targets a known weakness in video multi-modal large language models (MLLMs): their tendency to produce correct answers while referencing irrelevant visual information [1][2]. Existing reinforcement learning frameworks for evidence grounding typically use Intersection over Union (IoU) rewards, which measure pixel-level bounding box overlap. These geometry-only signals can be sensitive to boundary perturbations and fail to capture whether the cited evidence is semantically relevant to the question [1][2]. SER reformulates the grounding task as a constrained verification problem. Instead of computing geometric overlap, the method deploys a secondary vision-language model (VLM) as a referee. This local checker evaluates model-generated evidence claims along two axes: relevance and localization quality, and applies a temporal penalty for incorrect frame selection [1][2]. The design reduces dependence on dense box annotations, allowing training to proceed directly on standard video question-answering datasets [1][2]. On the V-STAR benchmark, SER achieved a 49.6% mLGM score, a 3.0-point improvement over the evidence-grounded Open-o3-Video baseline [1][2]. The paper was posted on arXiv, an open-access repository for electronic preprints that, as of late 2024, receives approximately 24,000 submissions per month across fields including computer science and electrical engineering [6]. The work appears within the Computer Vision and Pattern Recognition subject area, a domain where community-developed tools such as the arXiv Bibliographic Explorer and Connected Papers are frequently used to navigate citation networks and discover related research [4][5].