EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

A new benchmark called EComAgentBench tests how well AI shopping agents handle long-horizon tasks where a buyer's requirements are scattered across a visible query, a tool-gated profile, and scripted clarification, according to a paper posted to arXiv [1]. The benchmark comprises 662 tasks grounded in real Amazon products and reviews [1]. Agents must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls [1]. Typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source [1]. Construction is automated, with every answer fixed in code before any text is generated and every sample validated [1]. An evaluation of seven models found that even the strongest attains only 57.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources [1]. The paper argues that existing benchmarks, which expose full intent upfront and grade only the final choice, cannot pose this long-horizon challenge or explain which requirement an agent missed [2]. The work arrives as large language models, defined as models with many parameters trained with self-supervised learning on vast amounts of text, underpin a growing number of production systems [7]. Companies such as DeepSeek, a Chinese AI firm founded in July 2023, have released open-weight models that rival proprietary offerings at a fraction of the reported training cost [6]. DeepSeek's V3 model was trained for a claimed US$6 million, compared with the US$100 million cost reported for OpenAI's GPT-4 in 2023 [6]. The EComAgentBench paper appears on arXiv, a preprint repository that has integrated with Hugging Face Spaces to make machine-learning research more accessible [3][4]. Through that integration, users can find open-source demos linked to papers in computer science, statistics, and electrical engineering and systems science categories [4]. Demos are built with tools such as Gradio and Streamlit and allow anyone with a browser to try models without writing code [3]. Researchers can link a Space to an arXiv paper by including the paper's URL in the Space's README file or by associating a model on the Hugging Face Hub with the Space [5]. The benchmark's authors frame EComAgentBench as a reproducible foundation for moving shopping agents from single-query search toward dependable assistance over long horizons [2].