VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA

A new dataset called VinQA pushes multimodal document question-answering beyond text-only responses by requiring models to interleave cited charts, tables, and diagrams directly into long-form answers, according to research submitted on 15 Jun 2026 [1]. The work addresses a gap in how multimodal large language models, or MLLMs, handle real-world documents. While such documents routinely mix text with visual elements in varied layouts, existing QA research has largely produced text-only answers, underusing those visuals [2]. VinQA is grounded in relevant document pages and demands that generated answers explicitly cite and embed visual components alongside supporting text [1]. To feed raw page images into an MLLM, the researchers study two encoding strategies. Page Encoding encodes full-page images and treats bounding-box regions as citable units. Modality Encoding parses each page, extracts text, crops visual elements, and encodes them separately, using the cropped items as citable units [2]. The team also introduces M-GroSE, a multimodal evaluation framework that extends GroUSE to score answers on four dimensions: completeness, answer relevancy, faithfulness, and unanswerability. A separate metric, Visual Source F1, directly measures visual citation accuracy [2]. Proprietary frontier models still post the highest overall scores on the VinQA test split. However, fine-tuning the open Qwen2.5-VL models — a family of large language models developed by Alibaba Cloud and distributed under open-source licenses [9] — on the training split substantially lifts their performance and narrows the gap with proprietary systems [1]. Modality Encoding proves more robust initially for complex documents containing long text, many visual elements, and diverse citation needs. After training on VinQA, Page Encoding reaches a comparable level, competing effectively without the explicit parsing that Modality Encoding requires [2]. An MLLM-based judge called Visual G-Eval confirms that the fine-tuned models insert visual elements at semantically appropriate positions with faithful supporting text [2]. The paper appears on arXiv, a preprint server that accounts for roughly 95 percent of the paper URLs Hugging Face users link in their repositories [4]. Hugging Face and arXiv have collaborated to embed interactive demos directly alongside papers, allowing users to try state-of-the-art research without writing code [5].