How to Train Your Long-Context Visual Document Model

Researchers have introduced a synthetic data pipeline for long-document understanding, achieving improved performance on benchmark tasks. The pipeline generates thinking traces by scoring each page for question relevance and extracting textual evidence^[1].

A comprehensive study on training long-context visual document models has achieved state-of-the-art performance on MMLongBenchDoc, a benchmark for long-document visual question answering. The study systematically examined continued pretraining, supervised finetuning, and preference optimization for 24B and 32B parameter models, achieving state-of-the-art performance on MMLongBenchDoc for both parameter scales^[1]. The researchers also released MMLBD-C, a manually corrected version of MMLongBenchDoc, to reduce erroneous and low-quality examples in the benchmark. The synthetic data pipeline introduced by the researchers generates thinking traces by scoring each page for question relevance and extracting textual evidence. The resulting reasoning capability is internalized via low-strength model merging. The study found that training on context lengths that match evaluation context lengths outperforms training on longer contexts. The accuracy on MMLongBenchDoc with Qwen3 VL was 58.3, while the points improvement over distillation from Thinking version's traces on MMLBD-C with Mistral was 3.8^[1].