Modeling semantic association in self-paced reading with language model embeddings

A new study examining how the brain processes written language finds that the choice of language model used to measure semantic association can significantly alter conclusions about reading difficulty, with sentence-level embeddings proving more reliable than older word-level methods. The research, led by Sara Møller Østergaard, analyzed a corpus of Dutch texts read by participants whose brain activity and reading speed were simultaneously recorded [1]. The study tested ten different implementations for calculating semantic association—the meaningful relationship between a word and its surrounding context—by varying the underlying embedding model and the length of context considered [2]. The findings were evaluated against two measures: the N400, a well-established neural marker of semantic processing, and self-paced reading times, a behavioral measure of processing difficulty [3]. Reading is a multifaceted process involving word recognition, comprehension, and fluency, and semantic association has been identified as a critical component even when a word's predictability is accounted for [7][2]. The study's results showed that uncontextualized word embeddings, such as word2vec, which have been used in prior sentence-processing research, showed no reliable effects on either the N400 or self-paced reading times in this analysis [3]. In contrast, implementations that relied on sentence embeddings consistently predicted processing difficulty beyond what word predictability alone could explain [1]. “The choice of embedding model can alter the estimated effect of semantic association on both the N400 and self-paced reading times,” the authors state, underscoring the fragility of findings in this domain [2]. The paper provides illustrative examples from the Dutch texts, noting that sentence embeddings appear to capture a thematically coherent representation of semantic association in natural reading [3]. This work arrives amid a broader effort to align language model representations with human cognitive signals. A separate study probing model representations for human reading times found that the predictive power of these models depends strongly on the reading-time measure, the model layer, and the language under study, rather than being captured by a single universal predictor [4]. Another recent investigation into garden-path sentences—complex constructions that initially lead readers astray—found that specialized cognitive recovery models outperformed GPT-2 surprisal in fitting human reading data, though adding surprisal still improved the model's fit [5]. The new findings highlight the importance of methodological transparency when using language models to study human cognition. As the authors conclude, the results demonstrate that “conclusions critically depend on how semantic association is implemented, particularly with respect to the embedding model” [3].