IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows

A new benchmark called IHBench measures how well voice agents recover after a user interrupts them mid-sentence, finding that closed-weight models from OpenAI and Google consistently outperform open-weight alternatives across 10 enterprise domains [1]. The benchmark, detailed in a paper posted to arXiv on June 17, evaluates 27 audio-language model configurations by injecting six distinct interruption types at controlled points while an agent is speaking [1]. Each interruption is scored on two axes: task fulfillment — whether the agent completes the workflow correctly — and recovery quality — whether it addresses the user’s interjection without repeating already-heard content [2]. Existing evaluations for speech-capable models have focused on barge-in detection and turn-taking timing, leaving unmeasured what happens after the interruption [2]. IHBench fills that gap by running agents through state-machine-driven workflows in domains such as customer service, healthcare scheduling, and account management [2]. The researchers found that closed-weight models degrade roughly 3.3 times more slowly as conversations grow longer and show no gap between audio and text modalities, whereas open-weight models lose ground on all three measures [1]. A human study validated the LLM judge used for scoring against human annotators, and a cross-benchmark analysis against AudioMultiChallenge indicated that recovery quality is a largely distinct capability axis [2]. Google and OpenAI, the two companies whose closed-weight models led the results, have both invested heavily in audio-capable AI. Google’s Gemini architecture is trained natively on multiple data types, allowing it to process and generate text, code, images, audio, and video simultaneously [9]. OpenAI’s GPT family of large language models, along with its DALL-E and Sora series, has influenced industry research and commercial applications since the release of ChatGPT in November 2022 [10]. The IHBench results arrive as AI research becomes increasingly industry-driven. A 2023 analysis found that Google, OpenAI, and Meta have been responsible for some of the largest training runs and developed a large fraction of the algorithmic innovations underpinning large language models [7]. The new benchmark adds a dimension — post-interruption recovery — that is not captured by existing evaluation paradigms, much as generative search introduced new dimensions around retrieval behavior and synthesis that traditional search metrics missed [4].