Pop-Up Distractions Reveal Bag-of-Events Behavior in Video Large Language Models

A new study finds that popular video large language models systematically confuse events from different parts of a video, frequently hallucinating interactions between entities that never co-occurred. The researchers call this “bag-of-events” behavior and warn it signals a fundamental weakness in temporal reasoning. The findings, submitted on 26 May 2026 to the arXiv preprint server, introduce a benchmark called DistractionBench to stress-test how well VideoLLMs link subjects to events across time [1]. The authors inserted short advertisement clips into longer videos and then queried the models about the content. They report that the models often attributed actions from the injected ads to subjects in the main video, fabricating interactions that never took place [1][2]. “We characterize this systematic hallucination as bag-of-events (BoE) behavior, where models process videos as collections of events rather than temporally structured sequences,” the paper states [2]. All 11 VideoLLMs evaluated exhibited substantial BoE behavior, suggesting the problem is endemic rather than limited to a single architecture [1][2]. The concept of a “bag-of-events” draws a parallel to the bag-of-words model in natural language processing, which discards word order. In the video domain, the models appear to treat a video as an unordered set of visual occurrences, losing the temporal scaffolding that tells a viewer which subject performed which action and when [2]. The researchers argue this indicates that current VideoLLMs lack reliable mechanisms for temporal grounding [1][2]. DistractionBench’s methodology of splicing unrelated content into a primary video mirrors real-world scenarios where users might ask a model to summarize a long recording that contains channel-surfing breaks, advertisements, or post-credits scenes. Films have long used mid- and post-credits sequences for comedic gags or sequel teases, and web series such as “The Guild” distributed episodes across YouTube, Xbox Live, and other platforms, often interspersed with promotional material [4][5]. A model that cannot disentangle these segments could produce summaries that confuse a show’s characters with actors from an unrelated commercial. The paper does not single out specific commercial models by name in its abstract, but the finding that every tested system showed the flaw underscores a gap between the impressive captioning and question-answering benchmarks VideoLLMs have achieved and their actual temporal reasoning capabilities [1][2]. The authors call for the development of architectures with more robust subject-event association, moving beyond treating video as a flat collection of detected objects and actions [2].