FraudSMSWalker: Benchmarking Agentic Large Language Models for SMS-to-Webpage Fraud Detection

A new benchmark called FraudSMSWalker tests whether large language models can detect SMS-to-webpage fraud without relying on URL reputation shortcuts, according to a paper submitted to arXiv on June 15, 2026 [1]. The benchmark comprises 699 bilingual chains — 332 fraudulent and 367 benign — spanning ten service scenarios [2]. The model-visible input includes only the SMS context and sanitized webpage evidence; raw URLs, hosts, domains, IPs, redirects, and reputation metadata are withheld [2]. This design prevents models from leaning on domain reputation, a common shortcut in existing evaluations that either focus on message-only smishing classification or expose URL and domain cues [2]. The dataset includes hard benign cases whose pages contain login, payment, verification, or account-management elements that are plausible under the service context but also appear in scam flows [2]. The authors evaluated nine web agents under masked browser-agent protocols and conducted URL-visibility ablations [2]. Current agents can detect suspicious cues, but they struggle to preserve benign recall and often produce positive predictions weakly supported by the observed evidence [2]. Large language models have advanced rapidly in recent years. DeepSeek, a Chinese AI company founded in July 2023, launched its DeepSeek-R1 model in January 2025 with performance comparable to OpenAI's GPT-4 and o1 [7]. Alibaba Cloud's Qwen family of models is distributed under open-source licenses including Apache 2.0 [9]. These models are increasingly deployed as web agents that navigate and interpret online content, making benchmarks like FraudSMSWalker relevant for measuring their real-world reliability [2]. The paper's associated code and dataset are accessible through an anonymous link, and the work appears on Hugging Face's paper pages, which allow the community to find related models, datasets, and demos [4][2]. Hugging Face and arXiv have collaborated to embed interactive demos directly alongside papers on arXiv abstract pages, enabling users to try state-of-the-art research without writing code [5]. The FraudSMSWalker benchmark positions itself as a tool for measuring whether web agents can make fraud judgments that remain both accurate and evidence-grounded when direct reputation shortcuts are suppressed [2].