MM-TRELLIS: Point-Cloud Guided Multi-Modal 3D Vehicle Generation in Autonomous Driving

A research team has introduced MM-TRELLIS, a new method for generating high-fidelity 3D vehicle models by combining multi-view camera images and LiDAR point clouds from autonomous driving datasets into native 3D generative models [1]. The approach, detailed in a paper submitted to arXiv on June 23, 2026, addresses a persistent limitation in autonomous vehicle simulation: existing vehicle generation methods fail to fully exploit multimodal sensors and often rely on neural rendering-based reconstruction, which produces low-quality meshes [1]. While native 3D generative models have advanced significantly, they are not designed for arbitrary multi-view inputs and frequently struggle with real-world driving imagery [1]. MM-TRELLIS, described as a multi-modal extension of the TRELLIS architecture, integrates both image and LiDAR sensors from autonomous driving datasets directly into the generation pipeline [1]. Multi-view images serve as conditioning inputs, while LiDAR point clouds provide test-time guidance to enforce geometric accuracy and cross-view consistency [1]. During the denoising process, the method aligns the guidance point cloud with model priors and then enforces consistency between the generated geometry and the guidance data [1]. A voxel filtering strategy based on the opacity of 3D Gaussian Splatting is also introduced to suppress floating artifacts and produce clean meshes [1]. The researchers conducted comprehensive experiments on the Waymo dataset, demonstrating that MM-TRELLIS outperforms existing methods in high-fidelity 3D vehicle generation [1]. The code for the project has been made publicly available on GitHub [1]. Recovering realistic 3D vehicle models from autonomous driving scenes is considered crucial for synthesizing training data and building simulation environments, applications that demand geometric precision and visual fidelity [1]. The integration of LiDAR guidance at test time represents a departure from methods that rely solely on image-based reconstruction, offering a pathway to models that better reflect real-world sensor configurations [1]. The work builds on the broader trend of incorporating multiple data modalities into generative frameworks, a strategy that has shown utility in other domains where complementary sensor inputs improve model robustness [4]. The paper appears on arXiv under the Computer Vision and Pattern Recognition category [1].