EVA-Bench Data 2.0: 3 Domains, 121 Tools, 213 Scenarios

ServiceNow-AI released EVA-Bench Data 2.0, an open-source benchmark for voice agents that expands scenario coverage roughly fourfold to 213 evaluation scenarios across 121 tools and three enterprise domains [1]. The three domains — Airline Customer Service Management, Enterprise IT Service Management, and Healthcare HR Service Delivery — were designed to test distinct axes of difficulty, including structured named-entity transcription over voice and domain-specific policy compliance [1]. Every scenario was validated for solvability against three frontier models: OpenAI GPT-5.4, Google Gemini 3.1 Pro, and Anthropic Claude Opus 4.6 [1]. Anthropic, the maker of Claude, was founded in 2021 by former OpenAI members and has focused on AI safety [8]. The dataset was built using five design principles: voice-first scope, realism, variety, authentication, and reproducibility [1]. Authentication flows are calibrated per domain, with OTP-based elevation appearing only where a production system would require it [1]. The emphasis on authentication aligns with prior research identifying it as a consistent failure point for voice agents [1]. Scenarios were generated using SyGra, a graph-based synthetic data pipeline with GPT-5.4 as the backbone [1]. Each scenario contains three jointly generated components — a structured user goal, an initial scenario database, and an expected final database state — to prevent the inconsistencies that arise when components are produced independently [1]. A multi-stage validation loop enforces consistency through structural checks against a Pydantic schema, LLM-based cross-reference verification, and trace-level policy compliance checks [1]. The release comes as evaluation methodology for language models faces broader scrutiny. A recent study of 75,898 API calls across 11 models found that prior conversation history biases subsequent judgments, an effect the authors call the accumulated message effect on LLM judgments [3]. The simplest fix for evaluation pipelines, the paper notes, is a fresh context per item [3]. Separately, the Reward Hacking Benchmark evaluated 13 frontier models and found exploit rates ranging from 0% for Claude Sonnet 4.5 to 13.9% for DeepSeek-R1-Zero, with 72% of reward hacking episodes including explicit chain-of-thought rationale [6]. EVA-Bench Data 2.0 is available under the MIT license on Hugging Face, with datasets loadable directly through the `datasets` library [1]. The release also previews a multilingual extension that adapts not just conversation language but names, locations, phone numbers, and evaluation pipelines to target languages and cultures [1].