Capacity, Not Format: Rethinking Structured Reasoning Failures

Requiring large language models to output structured JSON can sharply reduce accuracy when the model is already operating near its performance ceiling, according to a new study that tested four models across five benchmarks. The paper, posted to arXiv on June 8, 2026, finds that the penalty imposed by structured output formats is not uniform across models but depends on what the authors call “spare capacity” [1]. When a model has sufficient headroom, JSON constraints cause little to no degradation. On the MATH-Hard benchmark, Sonnet scored 88.7±4.0% under JSON formatting compared with 89.3±1.7% using chain-of-thought prose, a difference the researchers describe as negligible [2]. For models pushed closer to their limits, the effects are severe and operate through two distinct mechanisms [1]. Under standard token budgets, Haiku’s performance fell by 36.2 percentage points (p < 0.0001), largely because responses were truncated before the model could complete its reasoning [2]. Even when token budgets were extended to eliminate truncation, GPT-4o-mini still dropped 28.0 percentage points (p < 0.001), a decline the authors attribute to pure capacity competition — the model’s computational resources being diverted from reasoning to format compliance [2]. The format penalty also scales with schema complexity and cannot be explained by prompt length alone (McNemar p < 0.0001) [2]. The findings challenge assumptions that frontier models are immune. On the AIME competition math benchmark, Opus 4.7 fell from 96.2% to 91.0% under JSON constraints, an exact difference of 5.3 percentage points [2]. Large language models are a type of machine learning system trained on vast text corpora to perform natural language tasks such as generation and reasoning [7]. The study’s authors used information-matched prose controls and a four-level schema complexity gradient to isolate format-specific effects from confounds introduced by longer prompts [2]. A delayed-structure ablation — in which the model reasons freely before formatting its answer — recovered most of the lost accuracy, with a three-run mean of 80–87% [2]. The result supports the capacity competition mechanism and points to a practical mitigation: when a model is near its limits, it is better to let it think first and format later [2]. The paper’s code and data are available through Hugging Face, a platform for sharing machine learning models and datasets [6].