Benchmarking Agentic Review Systems

Agentic review systems powered by large language models can track human quality judgments and catch errors in research papers, according to a new benchmarking study that tested multiple systems across frontier AI models [1]. The study, posted to arXiv on June 18, evaluated two open-source systems — OpenAIReview and coarse — alongside one proprietary system, Reviewer3, and a zero-shot baseline, running them across six large language models [1]. Large language models are neural networks trained on vast amounts of text for tasks including generation, summarization, and analysis [3]. Agentic AI systems, a subset of intelligent agents, proactively pursue goals and make decisions over extended periods [4]. On a pairwise accuracy test using ICLR and NeurIPS papers, every system performed above chance. The top configuration, OpenAIReview paired with GPT-5.5, achieved 83.0% accuracy when measured against external quality signals such as citations and acceptance decisions [1]. OpenAI developed the GPT family of models and has influenced industry research through releases including ChatGPT [6]. To assess error detection, researchers constructed a perturbation benchmark that injected four categories of errors into papers across eight arXiv subject classes. The strongest configuration, again OpenAIReview with GPT-5.5, caught 71.6% of injected errors [1]. When detections were combined across all six models, the union recall rose to 83.3%, indicating that different models catch different errors and that improved system design could boost performance further [1]. Language model benchmarks are standardized tests that measure capabilities including reasoning, factual accuracy, and generation quality [5]. A public deployment of OpenAIReview with real users showed that votes on its comments skewed positive at 1.44 to 1, though the most common complaints involved false positives and minor nitpicks [1]. The study concludes that while AI reviews still have room for improvement, they can already track human quality judgments, catch important errors, and earn positive feedback from real users [1]. Google's competing Gemini model family, which powers its chatbot and assistant products, has also focused on enhancing agentic capabilities for autonomous research tasks [8].