3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models

A new large language model, 3D-PLOT-LLM, can identify and reason about the individual parts of three-dimensional objects, a capability its creators say is absent from existing 3D multimodal models that describe objects only as a whole [1][2]. The model, detailed in a paper posted to the arXiv preprint repository on June 18, reorganizes the input token stream so that object parts become directly addressable through the language model's own vocabulary [1][2]. It partitions the frozen point encoder's patches into locally coherent regions and inserts a learnable marker and a reserved vocabulary token before each region's patch tokens [2]. A Marker-Space Refinement module then conditions each marker on its region's spatial statistics and adjacency neighbors [2]. The result is a model that can cite parts in its output and follow prompts that refer to parts by token, a capability the researchers describe as absent from prior object-level 3D MLLMs [2]. To evaluate this part-level interface, the researchers constructed PartVerse-QA, a vocabulary-level question-answering benchmark adapted from PartVerse mesh annotations, comprising 77,000 training pairs and 588 held-out queries on disjoint object splits [1][2]. On this benchmark, 3D-PLOT-LLM achieved a caption-to-slots Jaccard score of 0.459 and an exact-match score of 13.78%, with a slot-to-caption GPT-4o judge score of 44.68 [1][2]. On the 3DCoMPaT-GrIn part-aware grounded description benchmark, the model outperformed PointLLM, Kestrel, PARIS3D, and SegPoint on every text-output metric, and ShapeLLM on three of four, with a GPT-4o judge improvement of up to +3.03 over PointLLM [1][2]. On the Objaverse whole-object captioning benchmark, adding PartVerse-QA during a second training stage yielded a +0.65 SBERT score and +1.85 GPT-4o score over PointLLM [1][2]. The model also topped PointLLM-PiSA on four of five traditional metrics—SBERT, SimCSE, BLEU-1, and METEOR—despite targeting a part-grounded objective rather than whole-object description [1][2]. The researchers note that all of this was achieved with fewer than one million new trainable parameters on a frozen point encoder, an order of magnitude below prior part-aware 3D MLLMs, and without a segmentation decoder or bounding-box head [1][2]. The paper appears on arXiv, an open-access repository for electronic preprints in fields including computer science, which as of November 2024 receives about 24,000 submissions per month [7]. The work represents a departure from earlier part-aware approaches that added segmentation decoders, heavier 3D encoders, or bounding-box grammars at substantial parameter cost [2]. Large language models, the broader class of machine learning models to which 3D-PLOT-LLM belongs, are trained with self-supervised learning on vast amounts of text and are designed for natural language processing tasks such as language generation [9].