MMClima: A Framework for Multimodal Climate Science Data and Evaluation

A new multimodal climate question-answering framework, MMClima, has been released on arXiv, providing over 104,000 expert-validated question-answer pairs to benchmark AI models on climate science tasks [1][2]. The framework, detailed in a paper submitted on June 8, 2026, by Muhammad Umer Sheikh and colleagues, addresses a gap in existing climate AI benchmarks, which the authors describe as small, mostly textual, and covering a narrow range of models [1][2]. MMClima spans articles, video transcriptions, and scientific figures across five core climate science domains [2]. The dataset was constructed through automated claim extraction and question-answer synthesis, followed by human-in-the-loop validation to ensure reliability at scale [1][2]. Using the new benchmark, the researchers evaluated state-of-the-art multimodal language models on tasks requiring factual recall, visual interpretation, and cross-modal synthesis [1][2]. Large language models are neural networks trained on vast amounts of text and are foundational to modern chatbots, though their outputs can be unreliable if training data is biased or inaccurate [8]. The team also fine-tuned a model on the textual portion of the dataset, producing mmclima-70b-txt, a domain-adapted baseline that outperformed strong open- and closed-source models on textual question-answering [1][2]. The paper was posted to arXiv, an open-access repository for electronic preprints that has been in operation since 1991 and now receives approximately 24,000 submissions per month [6]. The submission history lists the article size as 8,788 KB [1]. The authors have released the dataset, evaluation pipeline, fine-tuned model weights, and data creation framework to support standardized multimodal evaluation for climate science [2]. The work appears within arXiv's broader ecosystem of community-developed tools. arXivLabs, a framework launched in 2020, allows collaborators to develop and share experimental features directly on the repository's website under guidelines that emphasize openness, community, excellence, and user data privacy [5]. Current Labs projects include the Bibliographic Explorer for navigating citation trees and the CORE Recommender for discovering related open-access papers [4][5].