Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams

Researchers have introduced new datasets and frameworks to advance spatial intelligence and reasoning in egocentric video streams, a key area for developing AI assistants.

The UCS-Bench dataset, spanning over 170 hours of egocentric visual observations with 8.1K+ timestamped questions, has been introduced to diagnose User-Centric Continual Spatial intelligence^[1]. Alongside this dataset, the DirectMe framework has been proposed to incrementally construct and maintain a structured spatial memory from streaming egocentric observations. DirectMe enables robust tracking and recall of object locations relative to the user's movement over time. In a separate development, researchers have introduced S-Agent, a spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos^[2]. S-Agent reshapes spatial perception into scene-centric understanding and uses a hierarchy of spatial tools and experts to ground objects in 2D, lift them into 3D geometric evidence, and aggregate this evidence into high-level spatial knowledge. Both DirectMe and S-Agent have shown significant improvements in spatial reasoning for leading multimodal LLMs and VLMs.