A Cross-Model VLM-Judge Protocol for Single-Image 3D Mesh Quality (and Why Cheap Proxies Fall Short)

Researchers have proposed a new evaluation protocol for assessing the quality of 3D meshes generated from single images, using a fixed 24-view render rig and two independent vision-language judge families.

The proposed protocol aims to address the lack of a standardized method for evaluating the quality of 3D meshes generated from single images. According to the researchers, the protocol uses a fixed 24-view headless render rig and two independent vision-language judge families to assess the quality of the generated meshes^[1]. The two judge families were found to agree substantially with each other, with a Cohen's kappa of 0.66. The researchers also tested the protocol with six adaptation methods and found that the most targeted adaptation method reached parity with the base model, with win-rates being directional at n=8 objects^[2]. In contrast, the researchers found that cheap automatic proxies, such as render-CLIP similarity and mesh geometry-validity statistics, do not substitute for the proposed protocol. Geometry validity was found to be only a weak signal on average and stayed below the pre-registered target, while render-CLIP was at chance^[1].