IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models

A team of researchers has introduced IdealGPT, a framework that uses large language models to iteratively break down vision-and-language reasoning tasks, achieving marked gains over existing models on two benchmark datasets. The framework, described in a paper by Rui Sun and colleagues, tackles what the authors identify as key weaknesses in prior divide-and-conquer approaches to vision-and-language understanding. Earlier methods, they argue, depend on domain-specific models to decompose questions and often force a final answer even when the intermediate information is incomplete [1]. IdealGPT instead employs three modules in a loop: one large language model generates sub-questions, a vision-language model supplies sub-answers, and a second large language model reasons toward a final answer. The cycle repeats until the system reaches a confident conclusion [1]. The researchers evaluated IdealGPT on two challenging zero-shot reasoning benchmarks. On the Visual Commonsense Reasoning dataset, the framework delivered an absolute improvement of 10% over the best existing GPT-4-class models. On the SNLI-VE dataset, the gain reached 15% [1][2]. The paper was first submitted to the arXiv preprint server on May 24, 2023, and was most recently revised on June 18, 2026 [1]. While the primary paper focuses on vision-and-language reasoning, the broader context of large language model iteration has drawn attention across scientific domains. For instance, transfer learning strategies that leverage large pre-trained datasets have been explored to improve model performance on smaller, specialized datasets in fields such as catalysis informatics [4]. Those efforts highlight a parallel challenge: ensuring that models trained on one distribution can generalize or be fine-tuned effectively for related tasks, a concern that also underpins the zero-shot evaluation setting used for IdealGPT [1][4]. The paper’s code has been made publicly available, and the work has been indexed by several academic discovery tools, including CatalyzeX and DagsHub, which help researchers locate code and data associated with publications [3][5]. The iterative reasoning design of IdealGPT represents a step toward models that can assess the sufficiency of their own intermediate outputs before committing to an answer, a capability the authors frame as a direct response to the premature prediction problem they identify in earlier systems [1][2].