Phonological Perception of Sign Language Models

Deep-learning models trained to recognize American Sign Language develop an emergent sensitivity to the language's phonological building blocks, but the type of sensitivity depends on the model's architecture, according to a study submitted in 2026 [1]. The research, posted to arXiv, examined Sign Language Recognition (SLR) models to determine whether they distinguish abstract phonological features — such as handshape, location, and movement — or simply exploit low-level statistical patterns [1]. Sign languages, like spoken languages, are compositional systems in which meaning arises from combining sublexical parameters [2][6]. In ASL, for instance, the signs CAR and WHICH differ by handshape, while APPLE and ONION contrast by location [6]. The study probed SLR models trained on ASL using minimal pairs and measured how closely the models' internal representations aligned with human behavioral data [1][2]. The findings showed a functional split. Pose-based models, which track skeletal keypoints, proved more sensitive to fine-grained handshape contrasts. Pixel-based models, which process raw video frames, better captured broader spatial distinctions such as sign location [1][3]. "This functional split suggests that current training methods and architectures are insufficient for holistic sign language processing," the authors wrote, adding that a more complete model likely requires integrating both skeletal abstraction and holistic spatial processing [2][3]. The pose-based models also produced latent representations that correlated with human perceptual similarity judgments, with a coefficient of approximately r~0.49 [1][2]. This alignment partially reproduced the confusion patterns and hierarchical groupings observed in human signers [2][3]. The study builds on earlier work that explicitly incorporated phonological knowledge into sign recognition. A 2023 paper found that training models to predict phonological characteristics alongside the sign itself yielded a nearly 9% absolute improvement in recognition accuracy on the WLASL benchmark [4]. The new findings underscore that phonological structure can emerge even without explicit supervision, though it remains fragmented by architectural design [1][3]. The concept of phonology in sign languages has been recognized for decades, treating components such as handshape and location as analogues to the phonemes of spoken languages [6]. The study's authors argue that cognitively grounded benchmarks are needed to guide the development of robust computational sign systems [2][3].