Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation

A new study finds that speech-aware large language models struggle to identify speakers, but a lightweight augmentation can bring their performance close to dedicated verification systems. Researchers led by Thomas Thebaud have benchmarked recent speech-aware large language models (LLMs) for speaker verification and found them lacking. The team proposed a model-agnostic scoring protocol that uses confidence scores or log-likelihood ratios from Yes/No token probabilities to produce continuous verification scores for both API-only and open-weight models [1]. On the VoxCeleb1 dataset, these LLMs showed weak speaker discrimination, with equal error rates above 20% [1]. Speech-aware LLMs are built on transformer architectures, which were introduced in the 2017 paper "Attention Is All You Need" and have since become the foundation of large language models [11]. These models can accept speech inputs, but their training objectives emphasize linguistic content or paralinguistic attributes such as emotions or gender, leaving it unclear whether they encode speaker identity [1]. To address this gap, the researchers introduced a lightweight augmentation that equips an LLM with automatic speaker verification capability. The method injects frozen speaker embeddings from an ECAPA-TDNN model through a learned projection and trains only low-rank adaptation (LoRA) adapters [1]. The ECAPA-TDNN is a neural network architecture designed for speaker recognition, and neural networks more broadly are computational models inspired by biological neural systems that learn hierarchical representations through layers of artificial neurons [3]. When applied to TinyLLaMA-1.1B, the resulting ECAPA-LLM achieved a 1.03% equal error rate on the VoxCeleb1-E test set [1]. This result approaches the performance of a dedicated speaker verification system while preserving the model's natural-language interface [1]. The paper was posted on arXiv, an open-access repository for electronic preprints that has hosted over two million articles since its launch in 1991 [9].