When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search

A new benchmark called DiscoBench has been introduced to test whether AI search agents can recognize vague or incorrect queries and ask clarifying questions, rather than blindly pursuing flawed search paths [1][2]. The benchmark, detailed in a paper submitted to arXiv on June 26, 2026, addresses a persistent weakness in large language model (LLM)-powered search tools. Existing evaluations typically assume users submit complete and explicit queries, but real-world requests are often underspecified or factually wrong [1][2]. In deep search scenarios, such ambiguity can cascade through multi-step reasoning chains, steering agents toward incorrect trajectories [2]. DiscoBench contains 211 samples and 463 ambiguity instances drawn from 11 real-world domains, covering four distinct ambiguity types [1][2]. The researchers also built a user simulator to enable multi-turn interaction, allowing them to measure performance across four dimensions: task utility, ambiguity detection, interaction strategy, and cost efficiency [2]. Experiments on representative LLMs revealed that the ability to detect ambiguity and the ability to ask effective clarification questions are separate skills. The paper reports that repeatedly searching instead of requesting clarification often yields worse results than direct guessing, underscoring a gap between raw retrieval capability and interactive problem-solving [1][2]. The work arrives as search agents become more common for complex information-seeking tasks that require multiple retrieval steps and reasoning [2]. arXiv, where the paper is hosted, is an open-access repository that has grown to a submission rate of about 24,000 articles per month as of November 2024 [8]. The platform also supports community-built tools through arXivLabs, a framework launched in 2020 that lets collaborators develop and share experimental features directly on article pages, such as citation explorers and code finders [6][7]. DiscoBench’s authors argue that closing the clarification gap is essential for making search agents reliable in practical settings where user intent is rarely perfectly stated from the outset [1][2].