What's the Point? Spatial Grammar & Index Resolution for Sign Language Processing

A new study proposes a computational framework to address a long-standing gap in sign language processing: the failure of current models to capture spatial indexing, a grammatical device that constitutes 10-15% of signing content [1]. The research, submitted on 6 Jun 2026, targets a specific non-lexical construction in sign languages where pointing gestures assign discourse entities to locations in the signing space for later reference [1]. Current sign language recognition models are predominantly trained with gloss-sequence or text supervision, which the authors argue leads to the systematic under-modeling of such productive constructions [1]. The framework decomposes spatial reference resolution into two stages: an Index Proposal Network that identifies indexing segments, and a differentiable online Entity Linking Module that incrementally clusters detected mentions into entity representations [3]. This modular design separates local recognition from discourse-level tracking [3]. Spatial indexing is a form of deixis, a universal linguistic feature where words or phrases refer to a particular time, place, or person relative to the context of the utterance [7]. In signed languages, this mechanism is fundamental to the visual grammar, enabling signers to “show while saying” by using the signing space to structure discourse [4]. Linguists distinguish between Fully Lexical Signs and Partially Lexical Signs, with pointing signs and depicting signs falling into the latter category [4]. Despite its prevalence, the study demonstrates that a strong continuous sign language recognition model fails to recover indexing tokens, even when its overall performance is competitive [3]. To reduce the need for large-scale manual coreference annotations, the researchers employ automatic cluster supervision derived from a large language model pipeline [3]. When integrated as inference-time biases into a frozen recognition backbone, the system reduces the Word Error Rate on indexing tokens from 96.2 to 59.6, while leaving the lexical Word Error Rate stable [3]. This result indicates that discourse-aware components can complement existing recognition systems without requiring retraining [3]. Coreference resolution in signed languages presents novel challenges because the meaning of pronominal signs is highly dependent on discourse and spatial context [5]. While coreference relations have been studied in sign linguistics, computational models have lagged behind [5]. The new framework establishes a baseline for index-aware sign language modeling, with the resulting mention representations enabling both automatic annotation and non-lexical structure modeling [1]. The authors indicate that future work will extend this treatment of spatial grammar to other productive constructions, including depicting signs and role shift [3].