DAR: Deontic Reasoning with Agentic Harnesses

Researchers have introduced Deontic Agentic Reasoning (DAR), a new approach to deontic reasoning that involves interacting with statutes on demand, and proposed the AARR benchmark series to evaluate agentic reasoning capabilities.

Deontic reasoning involves applying explicit rules and policies to case-specific facts, such as computing tax liability or determining the outcome of an immigration appeal^[1]. A key challenge for large language models (LLMs) is navigating long, cross-referenced rulesets. DAR addresses this by allowing models to interact with statutes as needed. DAR was evaluated on hard subsets of DeonticBench, with results showing that agentic harnesses can improve deontic reasoning performance, though not uniformly^[1]. Meanwhile, advancements in foundation models and agent scaffolding have enabled agents to demonstrate proficiency in complex coding tasks and autonomous experiment execution^[2]. However, frontier agents still exhibit limitations in field sensitivity, research ethics, and nuanced scientific judgment. The AARR benchmark series, which includes AARRI-Bench, aims to assess human-like professionalism, thoroughness, and nuanced reasoning in agents. In AARRI-Bench, the best-performing configuration achieved a 68.3% success rate^[2].