Latent Spatial Memory for Video World Models

A new video-generation framework called Mirage stores 3D scene memory directly in a diffusion model’s latent space, sidestepping the pixel-level point clouds used by earlier systems and yielding large gains in speed and memory efficiency, according to research posted on arXiv [1]. Video world models — systems that build internal representations of an environment and simulate how it changes over time — have become a focus of generative AI research [3]. Most current models that enforce 3D consistency across frames rely on explicit point cloud memory built in RGB space. That approach requires repeated rendering and variational autoencoder encoding, and the round trip through pixel space discards features learned by the latent representation [1][2]. To address those limitations, the authors introduce latent spatial memory, a persistent 3D cache that holds scene information directly in the diffusion latent space and avoids pixel-space reconstruction altogether [1][2]. The resulting framework, Mirage, constructs its memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping [1][2]. In experiments, Mirage delivered up to a 10.57× speedup in end-to-end video generation and a 55× reduction in memory footprint compared with explicit 3D baselines [1][2]. The system also achieved state-of-the-art performance on the WorldScore benchmark and strong reconstruction quality on the RealEstate10K dataset [1][2]. The work arrives amid a broader surge in generative AI capabilities. Since the 2020s AI boom, text-to-video models such as Veo, LTX, and Sora have drawn wide attention, while world models have been deployed in robotics, autonomous driving, and interactive video generation [3][5]. By operating entirely in latent space, Mirage removes both the information loss of pixel-space reconstruction and the computational cost of repeated encoding and rendering, the researchers write [1][2].