Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War

A new benchmark called Age of LLM pits large language models against each other in a turn-based strategic war game, revealing that a nuclear-rush strategy dominates play while diplomacy is rarely consummated, according to a preprint posted to arXiv on Monday. The benchmark places two LLMs on a 13x7 grid with the goal of destroying the opponent’s base. It imposes three deliberate stressors: fog of war, a full diplomacy channel that allows messages, ceasefires and ultimatums while keeping uranium stockpiles secret, and a reliability dimension in which every turn must follow a strict JSON schema or the action is silently discarded [1][2]. The engine is private and each match uses a fresh random map seed and opponent, a design choice intended to reduce the data contamination that can affect public benchmarks [2]. Researchers ran 15 reasoning models through 54 matches totaling 5,258 actions [1][2]. The nuclear rush — racing to build a nuclear launcher and fire — proved the dominant strategy, appearing in 78 percent of games on the rules-coherent v0.11+ sub-corpus and 85 percent of games corpus-wide [2]. The paper describes the sole-launcher signature as “largely mechanical under secret-simultaneous launch rules, not a cognitive deterrence failure” [2]. Military conquest was rare but faster, averaging 12.3 turns compared with 18.9 turns for other strategies [1][2]. Diplomacy was prolific across the matches yet almost never consummated [1][2]. The authors also found that roughly 58 percent of illegal actions stemmed from fog-of-war or state-tracking errors, making the illegal-action rate a proxy for belief-tracking ability [2]. A fifth finding, which the paper labels exploratory, suggests a weak link between reliability and winning [2]. The corpus is small, unbalanced and not side-swapped, so the authors caution that the resulting model ranking is a preliminary descriptive view rather than a formal contribution [2]. They frame the turn-by-turn traces of actions and messages as a lens for studying how LLMs reason under adversarial uncertainty — including belief-tracking, spontaneous deception and per-model cognitive “personas” — and release the replay format, an isometric viewer and all replays; engine source is available on request [2]. Large language models are machine learning systems with many parameters trained on vast text corpora for natural language generation [8]. Recent entrants such as DeepSeek and Qwen have drawn attention for their cost-efficient training and open-weight releases [7][9]. The Age of LLM benchmark adds to a growing landscape of evaluation frameworks that probe capabilities beyond static question-answering, though the authors stress that the current dataset is too limited for definitive model comparisons [2].