From USD Scenes to Knowledge Graphs: Zero-Shot Ontology Grounding with LLMs

Large language models can automatically link objects in 3D simulation scenes to formal ontology classes, a step that has long required brittle, manually built dictionaries, according to a paper posted to arXiv on June 8 [1][2]. The work targets a persistent bottleneck in constructing knowledge graphs from Universal Scene Description (USD) scenes for robot task reasoning: grounding scene objects to ontology classes [1][2]. The authors frame LLMs as a zero-shot, training-free alternative to hand-crafted mapping tables that fail to generalize across asset sets [2]. In experiments on a kitchen scene containing 125 objects and using the SOMA-HOME Ontology, LLMs reached 90–96% exact-match accuracy when object names were descriptive [1][2]. Accuracy fell to 49–89% when names were abbreviated, still outperforming dictionary and embedding baselines [2]. When names were fully opaque, context-augmented prompting recovered up to 48% accuracy [1][2]. A feature ablation showed the models rely heavily on semantic cues in the scene graph, specifically sibling names and parent paths. Anonymizing those cues dropped accuracy to 0–6%, while using geometry alone yielded only 4–17% [1][2]. The results indicate that LLMs exploit linguistic structure rather than spatial or visual features to perform the grounding. Large language models are neural networks trained on vast text corpora for tasks including generation, summarization, and translation [8]. The paper’s findings add to a growing body of work examining whether such models can replace hand-engineered components in robotics pipelines. The preprint appeared on arXiv, an open-access repository that hosts e-prints across mathematics, physics, computer science, and related fields [6]. arXiv does not peer-review submissions but moderates them before posting [6]. The repository passed two million articles by the end of 2021 and currently receives about 24,000 new submissions per month [6]. The paper is listed under the Computer Science > Robotics category [1].