Recurrent Reasoning on Symbolic Puzzles with Sequence Models

A new benchmark called RecurrReason tests whether sequence models can solve symbolic logic puzzles under controlled difficulty, revealing sharp performance gaps across puzzle types and architectures [1]. The benchmark, introduced in a paper posted to the arXiv preprint repository on 19 April 2026, comprises four recurrent logic puzzles: Tower of Hanoi, River Crossing, Block World, and Checkers Jumping [1]. Each puzzle is governed by a single interpretable difficulty parameter N that ranges from 1 to 10, producing 10,817 unique puzzles and 285,933 moves in total [1]. All trajectories are generated via breadth-first search to ensure optimal solutions [2]. Researchers trained two Transformer families — an encoder-decoder model styled after T5 and a decoder-only model styled after GPT-2 — on instances where N equals 1 through 7, then evaluated them on held-out in-distribution puzzles and harder out-of-distribution (OOD) puzzles at N equals 8 through 10 [1]. The T5 variant, when fine-tuned from a pre-trained checkpoint, reached 97.27% validation accuracy on Block World and 81.00% OOD accuracy on the same puzzle [1]. On River Crossing, however, every model scored 0.00% across all conditions [1]. The results underscore a broader challenge in artificial intelligence research. AI systems have long pursued goals such as reasoning, planning, and knowledge representation, often using techniques like state-space search and formal logic alongside neural networks [3]. Transformer architectures, which underpin large language models, use attention mechanisms to capture long-range dependencies in data [5]. Yet the RecurrReason findings indicate that architectural choices matter more than raw scale for these symbolic tasks [1]. Pre-training transferred only to puzzles whose transition functions exhibit local structure [1]. The paper’s failure-mode analysis arrives as interest in systematic reasoning benchmarks grows. Organizations such as Google DeepMind have previously demonstrated neural networks capable of mastering board games and discovering algorithms through reinforcement learning [6]. RecurrReason adds a complementary lens by demanding not just a valid answer but a minimal, robust solution under controlled difficulty scaling [2]. The authors state that the code and dataset will be open-sourced upon acceptance [1].