HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration

A research team has introduced HAT-4D, an agentic framework that reconstructs 3D geometry, temporal dynamics, and physical interactions of multiple objects from a single video, according to a paper submitted to arXiv on June 26, 2026 [1][2]. The framework, detailed in a preprint on the open-access repository arXiv, is described as the first designed to lift monocular video into 4D multi-object interaction reconstructions [1][2]. Existing monocular 4D reconstruction methods have concentrated on isolated objects and frequently fail when confronted with the severe occlusions and complex dynamics that characterize multi-object interactions [2]. HAT-4D addresses this by integrating vision-language models with a multi-level human-in-the-loop feedback mechanism, which resolves depth ambiguities and interaction-induced occlusions during 3D generation and 4D propagation [2]. The authors state that this approach yields physically plausible assets without requiring expensive multicamera rigs [2]. The paper also introduces MVOIK-4D, an open-world benchmark for monocular 4D interaction reconstruction, accompanied by a multi-dimensional evaluation protocol focused on physical plausibility and temporal consistency [2]. Extensive experiments reported in the preprint show that HAT-4D achieves state-of-the-art performance on most evaluation metrics while maintaining competitive semantic alignment [2]. Ablation studies indicate that even a small amount of human feedback improves interaction reconstruction, and data produced by the framework effectively boosts baseline performance when used for fine-tuning [2]. The work appears on arXiv, a repository that hosts electronic preprints across fields including computer science and physics and that, as of late 2024, receives approximately 24,000 submissions per month [6]. The paper’s abstract page features arXivLabs, a framework launched in 2020 that allows community collaborators to develop and share experimental tools directly on the site [5]. arXivLabs projects operate under guidelines that require partners to share arXiv’s values of openness, community, excellence, and user data privacy [5]. The code and data for HAT-4D have been made available through a project website [2].