LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization

A new speech tokenization method called LM-SPT, which distills semantic information through speech resynthesis, outperforms existing approaches on automatic speech recognition and text-to-speech tasks, according to a paper posted on the arXiv preprint server [1]. The method, proposed by Deajin Jo in a paper submitted on 20 June 2025 and revised on 14 June 2026, addresses a core challenge in speech language models: producing discrete speech tokens that align well with language models while operating at manageable frame rates [1][2]. Previous tokenizers often generate token sequences much longer than their text equivalents, complicating integration with pretrained language models [2]. Some recent approaches apply uniform average pooling to reduce the token rate, but this can oversmooth content-rich regions and dilute structural information [2]. LM-SPT takes a different path. Instead of directly matching teacher and student features through pooling, it resynthesizes speech from semantic tokens alone and then minimizes the discrepancy between representations extracted from the original and resynthesized waveforms, using a frozen, LM-aligned speech encoder [2]. This indirect supervision avoids rigid temporal alignment and encourages dedicated semantic units that remain aligned with language models even at reduced frame rates [2]. The paper reports that LM-SPT consistently outperforms previous semantic-enhanced speech tokenizers when applied to speech language models for automatic speech recognition and text-to-speech, without sacrificing speech reconstruction fidelity at the codec level [2]. The work appears on arXiv, an open-access repository of electronic preprints that, as of November 2024, receives about 24,000 submissions per month and has hosted over two million articles since its founding in 1991 [6]. The repository is not peer-reviewed; papers are approved after moderation [6]. The LM-SPT manuscript is listed under the Computation and Language category and is available in PDF and HTML formats [1].