SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models

A new evaluation framework called SpeechEQ reveals that even the most advanced voice-based AI models struggle to interpret emotional and social cues in conversation, according to research published on arXiv. The benchmark tests how Speech-Language Models handle the paralinguistic signals essential for natural spoken interaction. The framework, introduced by a team of researchers, is designed to assess the sociolinguistic reasoning of Speech-Language Models (SLMs) [1]. It moves beyond prior evaluations that focused on isolated text or passive acoustic perception, instead requiring models to navigate complex, multi-turn dialogues [2]. The SpeechEQ dataset comprises 2,265 dialogues that probe 15 distinct Emotional Quotient (EQ) subscales, a structure grounded in the EQ-i 2.0 theory of human emotional intelligence [1][2]. The benchmark uses a metric called the Spoken EQ (SEQ) score, inspired by human EQ assessments, to measure performance [2]. Experiments conducted with the SpeechEQ framework exposed significant limitations in how current systems understand and apply paralinguistic cues through speech [1]. The study evaluated both traditional Speech Emotion Recognition models and newer end-to-end SLMs. While the end-to-end architectures generally outperformed cascaded systems—where separate components handle speech recognition and language understanding—the results were far from demonstrating true emotional awareness [1][2]. The research identified three specific failure modes that bottleneck current multimodal models. The first is a text-reliant "modality shortcut," where models default to processing transcribed text rather than fully leveraging acoustic information. The second is an alignment-induced "safety trap," and the third is "contextual amnesia," where models lose track of emotional context across the turns of a conversation [1][2]. These findings underscore the gap between simply recognizing emotion in a single utterance and engaging in socially aware dialogue. The SpeechEQ benchmark and its dataset have been made publicly available on Hugging Face, with a demo page also provided for further exploration [2]. The work highlights the critical bottleneck that paralinguistic social cues represent for natural human-AI communication as conversational systems increasingly move into spoken interaction [2].