CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

A new benchmark called CostBench reveals that leading large language model agents struggle to devise and adjust cost-optimal plans, particularly when conditions change. The evaluation framework, introduced by researchers including Jiayu Liu, tests economic reasoning in a travel-planning domain where tools carry customizable costs and dynamic disruptions occur [1][2]. Current evaluations of LLM agents emphasize task completion while often overlooking resource efficiency and adaptability, according to the paper posted on arXiv [1][2]. CostBench was designed to bridge that gap by measuring agents' ability to identify the cheapest sequence of actions and to replan when circumstances shift. The benchmark is situated in a travel-planning domain, where tasks can be solved through multiple sequences of atomic and composite tools, each assigned diverse and customizable costs [1][2]. It supports four types of dynamic blocking events, including tool failures and cost changes, to simulate real-world unpredictability and force agents to adapt in real time [1][2]. Evaluations of leading open-sourced and proprietary models on CostBench uncovered a substantial gap in cost-aware planning. Agents frequently failed to identify cost-optimal solutions even in static settings. GPT-5 achieved less than 75% exact match rate on the hardest tasks under static conditions, and its performance dropped by around 40% when dynamic conditions were introduced [1][2]. The findings suggest that while models can complete tasks, they often do so without regard for economic efficiency, a shortcoming that becomes more pronounced when the environment changes. The benchmark's focus on economic reasoning aligns with broader discussions about resource-aware AI. The United Nations Sustainable Development Goals, adopted in 2015, emphasize the connections between environmental, social, and economic aspects of sustainable development, though a 2025 UN report noted that only 35% of SDG targets were on track or making moderate progress [7]. While CostBench does not address sustainability directly, its emphasis on cost-optimal planning echoes the principle that intelligent systems should consider resource constraints alongside task completion. The benchmark's authors argue that diagnosing these weaknesses lays the groundwork for developing future agents that are both economically rational and robust [1][2].