Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce

A new two-agent simulation framework enables controlled testing of conversational shopping assistants by pairing an independent buyer agent with interchangeable responder architectures, according to research published on arXiv [1]. The framework holds the buyer agent constant across experiments, allowing direct comparison of responder designs on identical scenarios. The buyer is configured with personas, missions, and patience levels, while the responder integrates with a real e-commerce search API [1]. The evaluation used 2,011 conversations across 14 persona buckets [1]. The study reports four empirical findings. Rolling-window memory outperformed intent-extraction memory on all quality metrics and was 35% faster per query [1]. A systematic failure analysis of one responder version led to targeted fixes that cut failure and near-failure rates by 62% across the full dataset [1]. When researchers swapped the responder's large language model backbone from Gemini 2.5 to Llama 3.3 70B, performance dropped by 0.16 to 0.45 points, despite the architecture remaining identical [1]. The paper also documents a systematic philosophical disagreement between frontier LLM judges: Gemini rewards process correctness, while Claude demands concrete outcomes, even when both use the same evaluation prompt [1]. The work arrives as major AI developers continue refining evaluation methods for agentic systems. Google DeepMind, which develops the Gemini family of large language models, has contributed to research on multi-agent interactions and safety evaluations, including the Melting Pot suite for assessing tendencies such as cooperation [6][9]. Anthropic, the maker of Claude, has published responsible scaling policies that outline how it monitors dangers from AI systems and applies mitigations as models become more capable [6][7]. Generative AI tools, including chatbots such as ChatGPT, Claude, and Google Gemini, have seen rapid adoption since the AI boom of the 2020s, driven by advances in large language models based on the transformer architecture [3]. Companies across software development, healthcare, finance, and customer service have deployed these systems [3]. The simulation framework described in the arXiv paper offers a structured method for comparing architectures before deployment, using a buyer agent that remains fixed to isolate the effect of responder design changes [1].