Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistants

A new benchmark called Shopping Reasoning Bench evaluates how well AI models handle the complex, multi-turn reasoning required by conversational shopping assistants, revealing that even the strongest systems fall short of expert-level advice. The benchmark, introduced in a paper posted to arXiv on June 10, comprises 525 missions — 232 single-turn and 293 multi-turn — assessed against 10,863 importance-weighted binary rubrics written by retail domain experts [1][2]. The rubrics are organized under a taxonomy of five reasoning categories and fifteen subcategories, including preference refinement, trade-off analysis, and compatibility assessment [2]. Researchers tested nine models from three families — GPT, Claude, and Gemini — and found overall pass rates between 57 and 77 percent [1][2]. Performance dropped sharply on optional, above-and-beyond criteria: all models scored 13 to 29 points lower on those measures than on required ones [2]. As conversations progressed through multiple turns, model performance degraded by 4 to 18 points [2]. “Shopping reasoning is unique among language model applications,” the authors write. “Unlike factual question answering or verifiable code generation, it requires balancing subjective preferences, budget constraints, and cross-product trade-offs across multi-turn dialogue, capabilities absent from previous e-commerce and general-purpose benchmarks” [2]. Conversational shopping assistants already serve hundreds of millions of customers, yet no prior benchmark jointly evaluated the open-ended reasoning, domain expertise, and criterion-level quality that real shopping conversations demand [2]. The Shopping Reasoning Bench is designed to fill that gap, providing a standardized testbed that exposes where current models succeed — basic shopping assistance — and where they falter — delivering nuanced, expert-level guidance [1][2]. The paper’s findings underscore the difficulty of building AI systems that can sustain coherent, personalized advice over extended interactions. While models can handle straightforward product queries, they struggle when asked to weigh competing priorities or adapt recommendations as new constraints emerge mid-conversation [2]. The benchmark’s multi-turn structure and expert-authored rubrics make it a demanding evaluation tool for future assistant development [1][2].