HERO'S JOURNEY: Testing Complex Rule Induction with Text Games

A new benchmark called HERO'S JOURNEY has been introduced to test how well large language models can infer hidden rules and execute multi-step tasks in text-based environments, according to a paper submitted in 2026 [1]. The benchmark, detailed in a paper by Anshun Asher Zheng and submitted to arXiv on June 1, 2026, covers eight distinct tasks split across attribute and procedural induction families [1]. Each task is built around four structural rule forms and includes controllable lexical grounding and identifiability conditions, allowing researchers to isolate specific reasoning capabilities [1]. The framework requires agents to deduce hidden rules from demonstrations and then apply them through goal-directed episodes, a process that mirrors the structure of classic puzzles. Riddles, for instance, are broadly categorized as enigmas, which demand ingenuity for a solution, and conundra, which rely on punning [2]. The HERO'S JOURNEY tasks present a similar challenge, requiring a system to cross conceptual boundaries for a solution [2]. Evaluations of state-of-the-art LLMs on the benchmark showed evidence of rule induction, but the ability was limited and uneven across the eight tasks [1]. The study also identified an execution bottleneck, where the process of carrying out multi-step actions posed a greater challenge for models than variations in surface semantics, which had a minimal effect [1]. Researchers tested induction-specific steering methods and found they improved performance on attribute-based tasks but produced no reliable gains on procedural tasks, indicating that procedural induction remains an open challenge [1]. The work was developed within arXivLabs, a framework for experimental projects that allows collaborators to build new features on the arXiv platform [1].