Disentangling Perception and Reasoning in Multimodal LLMs via Reward Design

New research finds that visual perception, not logical reasoning, is the primary bottleneck in multimodal large language models, and proposes reward-design strategies to address it [1]. The study, posted to arXiv by researcher Omar Sharif and colleagues, examines how multimodal models handle tasks that require both interpreting images and reasoning about them [1]. The authors show that when images are replaced with simple textual descriptions, performance on algorithmic visual puzzles jumps by more than 20 points on average for Claude models [1][2]. This gap indicates that errors originate in the perception stage rather than in downstream reasoning [1]. The work evaluates six reward functions designed to induce visual grounding during reinforcement learning without chain-of-thought supervision [1][3]. The functions vary along three axes: how the final answer is scored, what intermediate output structure is rewarded, and whether visual grounding is explicitly rewarded through a dedicated tag [3][4]. One variant introduces an explicit tag inside the reasoning block and rewards unique visual descriptions, testing whether explicitly rewarding image-referent content improves grounding [3][4]. Training the model Qwen-2.5-VL-7B with the GRPO algorithm, the reward design produced long, structured reasoning chains that included self-reflection and visual references, yielding a 5.56-point gain over the base model [1][2]. The gains were uneven: no single reward improved performance across all categories, and rewards that used verifiable accuracy signals traded out-of-domain transfer for higher in-domain accuracy [1][2]. The findings align with a broader research push to decouple perception from reasoning. A separate framework called PRCO, or Perception–Reasoning Coevolution, splits the task into two roles — an Observer that generates an evidence caption and a Solver that predicts the answer — and trains them with separate reward signals [5]. That approach yielded average accuracy improvements of over 7 points across eight multimodal reasoning benchmarks compared to the base model [5]. Another training-free pipeline, described in a paper indexed on Hugging Face, uses a large language model to orchestrate high-level reasoning while interrogating a multimodal model for specific visual details on demand, reducing visually unfounded reasoning steps [7]. The study’s authors argue that perception-aware reward design offers a path forward, so that signals correct perception at its source rather than the reasoning that inherits its errors [1][2]. The first version of the paper was submitted on 1 January 2026 at 8,918 KB; a revised version followed on 15 June 2026 at 9,307 KB [1].