Which Models Perform Better in Inheritance Reasoning?

A team from PSL has benchmarked commercial and open-source large language models on Arabic Islamic inheritance reasoning, finding a clear reliability gap favoring commercial systems in the QIAS 2026 Shared Task. [1] The study, submitted to arXiv on 11 June 2026, evaluates how well different model families handle inheritance cases requiring legal interpretation, multi-step reasoning, and precise numerical computation under a unified prompting strategy. [1] The best performance was recorded by Gemini 2.5 Flash, which achieved a Mean Reciprocal Error of 0.989. [1] Commercial models proved stronger at identifying eligible heirs, applying exclusion rules, and maintaining consistency across reasoning steps, while open-source models showed greater instability, particularly in cases involving dependent legal decisions and fractional share adjustments. [1] The findings align with broader patterns observed in earlier iterations of the shared task. A separate evaluation of seven distinct LLMs on Islamic inheritance law found that models with explicit reasoning capabilities, such as Gemini 2.5 and o3, achieved accuracy rates of 90.6% and 93.4% respectively, while models without such capabilities scored markedly lower. [3] GPT-4.5 reached 74.0% accuracy, but open-source Arabic-focused models like ALLaM and Fanar fell below 50%. [3] The performance degradation at advanced difficulty levels was especially pronounced for Arabic-focused models: ALLaM achieved 58.0% accuracy on beginner cases but dropped to 27.8% on advanced ones. [3] Other teams have explored alternative approaches to the same challenge. One group applied a Chain-of-Thought prompting strategy using Claude 3.7 and GPT-4o, finding that incorporating a structured reasoning step lifted Claude 3.7's score from 0.67 to 0.81 and GPT-4o's from 0.63 to 0.74. [4] A separate entry employed a majority voting solution combining Gemini Flash 2.5, Gemini Pro 2.5, and GPT o3, reaching 92.7% accuracy and securing third place in the challenge. [5] That team also observed that fine-tuning GPT-4o improved its accuracy to over 84%, while fine-tuning Gemini Flash 2.5 caused a performance drop from 91.5% to 74.6%. [5] The difficulty of the domain stems from the need to handle corrective procedures such as redistribution and proportionate reduction, which require precise fractional adjustments. [3] Error analysis across multiple studies confirms that models struggle most with complex correction procedures and dependent legal decisions, even when basic reasoning appears sound. [4] The PSL team's work contributes to a growing body of evidence that structured reasoning substantially enhances LLM performance on complex Arabic legal reasoning tasks without requiring additional training or retrieval-augmented generation. [4]