Speech Codec Probing from Semantic and Phonetic Perspectives

A systematic analysis of widely used speech tokenizers finds they encode phonetic structure far more than lexical-semantic meaning, challenging assumptions about how these components bridge speech and large language models, according to a paper posted on arXiv [1][2]. Speech tokenizers serve as a critical interface, converting continuous audio signals into discrete units that large language models (LLMs) can process [1][2]. LLMs, which are machine learning models trained on vast amounts of text for language generation, rely on these tokenizers in multimodal systems [8]. The field has operated on the expectation that tokenizers preserve both semantic and acoustic information for downstream tasks [2]. However, the new research argues that the term "semantic" in speech processing does not align with linguistic lexical-semantic meaning, creating a mismatch between speech and text modalities [1][2]. Xuan Shi and colleagues evaluated the lexical-semantic and phonetic content encoded by several widely used speech tokenizers through three distinct probing tasks [1][2]. Their results indicate that current tokenizers primarily capture phonetic rather than lexical-semantic structure [1][2]. The findings carry practical implications for the design of next-generation speech tokenization methods, the authors note [1][2]. The code for the research has been released publicly on GitHub [1][2]. The paper was submitted to arXiv on 11 March 2026, with a revised version posted on 24 June 2026 [1]. arXiv, an open-access repository for electronic preprints, hosts scientific papers across fields including computer science and electrical engineering, and is not peer-reviewed [6]. As of late 2024, the repository was receiving about 24,000 new articles per month [6]. The platform also supports arXivLabs, a framework enabling community collaborators to develop experimental tools that appear on article record pages, such as the Bibliographic Explorer and CORE Recommender [4][5].