Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization

Researchers have proposed a new method, Reasoning-Conditioned Direct Preference Optimization (RC-DPO), to mitigate hallucinations in multimodal large reasoning models, which can produce unsupported or incorrect statements.

Multimodal Large Reasoning Models suffer from severe hallucinations, according to a study published on arXiv^[1]. Hallucinations are a significant issue in language models, particularly in healthcare applications, as they limit the reliability of these models^[2]. The new RC-DPO method models the Chain-of-Thought (CoT) as a condition for answer generation, effectively mitigating hallucinations and improving the reliability of the multimodal reasoning process^[1]. This approach contrasts with existing training-based methods that typically optimize hallucinations through response-level direct preference optimization (DPO), treating CoT and the final answer as a single output. By explicitly formulating a CoT-oriented preference term, RC-DPO promotes answer-supportive reasoning chain alignment. Researchers employed Monte Carlo Tree Search to discover visually grounded and logically consistent CoTs as positive samples and attention-guided CoT token pruning to construct negative ones, further improving optimization. Extensive experiments across various models and benchmarks demonstrated the effectiveness of RC-DPO in mitigating hallucinations.