SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

Researchers have introduced SEATauBench, the first agent-focused evaluation framework designed to measure AI agent performance in Southeast Asian languages, adapting the TauBench benchmark to five regional languages including Mandarin, Vietnamese, Thai, Indonesian, and Filipino [1]. The framework, detailed in a paper submitted on June 27, 2026, evaluates agents across three progressively localized settings that vary the language of user-agent interaction, tool specifications, and task domains [1][3]. The work addresses a persistent gap in AI evaluation, where agent capabilities in regional languages remain poorly understood despite rapid growth in AI development for Southeast Asia [2]. SEATauBench adapts the existing TauBench, originally designed to emulate dynamic conversations between a user and a language agent equipped with domain-specific API tools and policy guidelines [6]. The original benchmark includes domains such as retail and airline, where agents must follow complex rules about product returns, baggage allowances, and flight changes [5]. SEATauBench extends this framework by developing a structured, non-breaking translation pipeline that translates different interfaces an AI agent interacts with without breaking execution [3]. The evaluation tests agents in three settings: L2 Interaction, which isolates linguistic capability in user-agent conversation; L2 Tool, which tests the ability to use tools with non-English tool specifications; and L2 Domain, which evaluates performance when all task contexts are in the target language [3][4]. Across three recent models, researchers found that English agent capabilities transfer reasonably well when only the conversation language changes, but quality and robustness degrade sharply as more task contexts are localized, with the largest losses occurring in full domain adaptation [1][2]. These findings expose a gap between the growth of Southeast Asian evaluation resources and the readiness of current agents for sovereign AI deployment [3]. The region has seen prior benchmarking efforts such as BHASA, a holistic linguistic and cultural evaluation suite for Southeast Asian languages covering tasks across natural language understanding, generation, and reasoning [7]. Other initiatives include LoraxBench, which covers 20 Indonesian languages across six tasks, and Komodo-7B, a 7-billion-parameter model designed for Indonesian and 11 regional languages [8][9]. Despite these efforts, English-centric evaluations have continued to overestimate multilingual capabilities [4]. The SEATauBench paper, authored by Saksorn Ruangtanusak and colleagues, was submitted as a 1,190 KB preprint [1]. The data and code are publicly available on GitHub [1][2].