Disagreement-Based Cross-Model Routing for Implicit Video Question Answering

A new inference-time method routes difficult implicit video questions to a second model, improving accuracy on a benchmark where conventional self-consistency strategies fail, according to a paper submitted 31 May 2026 [1]. The approach, proposed by Durga Sandeep Saluru, targets the ImplicitQA benchmark, which tests whether models can infer answers from off-screen events, line-of-sight cues, and cross-shot spatial layout rather than explicitly shown content [1]. On this benchmark, a single frontier video large language model already operates near its accuracy ceiling, and the researchers observed that majority voting across repeated samples of the same model can hurt performance because the model's errors on hard questions are correlated [1]. The method requires no labels and no training [1]. It triple-samples a native-video model, Gemini 3.1 Pro Preview, at temperature zero, exploiting genuine sample-to-sample variance in its video-processing pipeline to identify the roughly 20% subset of questions where the three samples disagree [1]. Only that subset is routed to a second model from a different family, Claude Opus 4.8, which consumes uniformly sampled frames with adaptive thinking [1]. Claude is a series of large language models developed by Anthropic and released as an AI-based chatbot in March 2023 [6]. On the 1001-question validation set with public ground truth, the method improved average accuracy by +1.43 over the best single sample of the primary model [1]. Per-category gains were concentrated on Motion & Trajectory (+5.49), Inferred Counting (+3.45), and Vertical Spatial Reasoning (+1.82), the categories most dependent on cross-shot reference resolution [1]. The same pipeline applied to the held-out 172-question CVPR 2026 ImplicitQA challenge test set achieved 82.03 average accuracy and 79.71 macro average accuracy, a gain of +1.81 over the best single sample of the primary model, confirming the validation result on an independent split [1].