Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking

A new benchmark called WebStep reveals that web agents with nearly identical success rates can differ sharply in how they explore and execute tasks, according to research published on arXiv. The 1,800-instance framework tracks semantic state behind the graphical interface, exposing process-level failures that terminal evaluation misses [1][2]. The study introduces WebStep, a benchmark of 1,800 task instances with controlled difficulty and automatic semantic state tracking [1][2]. Each website in the benchmark exposes a deterministic semantic MDP alongside the GUI. The agent operates on the interface while the environment records high-level states and transitions in the background, enabling fine-grained analysis without manual annotation [2]. Three agents whose success rates cluster within 31-33% were tested. Process metrics showed divergence in exploration reach versus execution accuracy that outcome evaluation alone could not detect [1][2]. Decomposing performance by skill exposed opposite per-skill rankings hidden within the same website. On the Housing domain, OpenAI CUA outperformed Qwen3.5 by 23.7% on commit actions yet underperformed it by 15.6% on filtering, pinpointing a concrete skill to improve even within a single domain [1][2]. Bifurcation analysis localized the decisive error that loses the task and showed that this error is agent-specific rather than shared across models [1][2]. The differences widened as tasks grew harder: success rates were similar on easy tasks but separated sharply when exploration became more demanding [1][2]. Web agents are a class of AI systems that pursue goals, use tools, and take actions with varying degrees of autonomy, typically operating within human-defined objectives and constraints [3]. Standardized benchmarks have long been used to evaluate language models on tasks such as language understanding, generation, and reasoning, with metrics extending beyond accuracy to include throughput, energy efficiency, and bias [4]. The WebStep framework departs from this tradition by capturing the process rather than only the terminal outcome. OpenAI, the developer of the CUA agent tested in the study, is a San Francisco-based research organization known for the GPT family of large language models and the release of ChatGPT in November 2022, which helped catalyze the current AI boom [6][7]. The company has faced scrutiny over AI safety practices, with roughly half of its then-employed safety researchers leaving throughout 2024 [6]. The research community has increasingly called for evaluation methods that go beyond aggregate scores, and the WebStep authors argue their process-level analysis provides fine-grained, actionable insight into where and how each agent should be improved [1][2].