When Iterative RAG Beats Ideal Evidence: A Diagnostic Study in Scientific Multi-hop Question Answering

A controlled diagnostic study finds that iterative retrieval-reasoning loops can outperform even an idealized static RAG setup in scientific multi-hop question answering, with gains reaching 25.6 percentage points on a chemistry-focused benchmark [1]. The study, led by Mahdi Astaraki and submitted to arXiv, provides what the authors describe as the first controlled, mechanism-level diagnostic evaluation of whether synchronized iterative retrieval and reasoning can surpass a Gold Context RAG upper bound, where all oracle evidence is supplied at once [1][2]. Researchers benchmarked 11 state-of-the-art large language models under three regimes: No Context, measuring reliance on parametric memory; Gold Context; and Iterative RAG, a training-free controller that alternates retrieval, hypothesis refinement, and evidence-aware stopping [1][3]. The experiments used the ChemKGMultiHopQA dataset, isolating questions that require genuine retrieval [1][4]. Across the models tested, Iterative RAG consistently outperformed the Gold Context baseline. The performance advantage was especially pronounced for non-reasoning fine-tuned models [2][5]. The analysis indicates that staged retrieval reduces late-hop failures, mitigates context overload, and enables dynamic correction of early hypothesis drift — benefits that static evidence delivery cannot provide [1][6]. The authors note that the process of staged retrieval is often more influential than the mere presence of ideal evidence [3][4]. Despite the gains, the study identifies persistent failure modes. These include incomplete hop coverage, distractor latch trajectories, early stopping miscalibration, and high composition failure rates even when retrieval is perfect [1][2]. One striking finding quantified in the paper is that, on average, 87.3% of all incorrect answers fall into a composition failure category, meaning the system is not limited by the reach of its retriever but by the fragility of its reasoning [4][6]. The model essentially “stumbles at the finish line,” unable to faithfully extract the answer it has already found [4]. The work offers practical guidance for deploying and diagnosing RAG systems in specialized scientific settings and establishes a foundation for developing more reliable, controllable iterative retrieval-reasoning frameworks [1][5]. The code and evaluation results have been made publicly available [4][5].