Planning with the Views

A new framework dramatically improves the ability of visual language models to plan and reason in 3D space, a capability researchers call “view planning,” by using self-exploration and a technique termed view graph distillation [1]. The research, detailed in a paper posted to arXiv, identifies a critical shortcoming in frontier visual language models (VLMs): while they understand how a single camera action transforms a view, they fail to compose these transformations across multi-turn plans [1]. The team, led by Manling Li, probed 13 VLMs in ViewSuite, a 3D point-cloud environment built on real ScanNet scenes [1]. The planning gap widens as the distance between viewpoints grows [1]. To address this, the researchers designed an iterative framework that alternates self-exploration with view graph distillation [1]. The core insight is that all exploration trajectories, successful or not, collectively form a view graph that compactly captures how viewpoints connect across a scene [1]. Distilling this graph into diverse supervised tasks reshapes the policy distribution and overcomes the sparse rewards that stall pure reinforcement learning [1]. The approach boosted the performance of Qwen2.5-VL-7B on interactive view planning from 2.5% to 47.8%, surpassing GPT-5.4 Pro at 18.5% and Gemini 3.1 Pro at 21.4% [1]. The work represents a step toward VLMs that can actively reason in 3D environments, a challenge distinct from other planning domains. In robotics, for instance, path planning for object reconstruction often relies on computationally expensive sampling-based methods that evaluate numerous candidate views [4]. A separate study proposed a more efficient solution using a focus point in the most informative region, reducing computation time by a factor of 6.2 to 19.36 while maintaining statistically equivalent reconstruction coverage [4]. In medical robotics, a differentiable rendering-based framework for 3D transpedicular path planning achieved bipedicular planning success rates of 82% with synthetic data and 75% with cadaver data, exceeding a 2D-to-3D baseline’s 66% and 31% rates, respectively [3]. The VLM view-planning framework differs by emphasizing the model’s own reasoning over many sequential steps rather than optimizing a single geometric path. The paper’s authors note that self-exploration emerges as a promising path toward VLMs that can actively reason and plan in 3D space [1]. Code and data for the project are available on the project’s website [1].