3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding

Researchers have presented two new frameworks, 3D-RFT and MUSE, for advancing 3D scene understanding and authoring. 3D-RFT applies reinforcement fine-tuning to video-based 3D perception, while MUSE enables controllable 3D scene authoring via a multi-agent framework.

3D-RFT, presented in a paper on arXiv^[1], is the first framework to extend Reinforcement Learning with Verifiable Rewards (RLVR) to video-based 3D perception and reasoning. It achieves state-of-the-art performance on various video-based 3D scene understanding tasks, with 3D-RFT-4B outperforming larger models on 3D video detection, 3D visual grounding, and spatial reasoning benchmarks. Meanwhile, MUSE, detailed in another arXiv paper^[2], is a memory-grounded multi-agent framework for controllable 3D scene authoring. MUSE improves All-Goal success from 37.9 to 80.7 on full construction cases and achieves 49.6 All-Goal success on a 240-case editing test split^[2]. The framework also demonstrates a 99.9% preservation rate and only a 0.6% unintended change rate on the editing test split. MUSE uses Working, Scene, and Skill Memory for requirement-level state tracking, with an Architect, a Sculptor, and an Inspector working together to satisfy incremental requirements.