Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioning

A new image captioning framework that retrieves external articles to add context beyond visible pixels placed fifth in the ACM Multimedia EVENTA 2025 Challenge, according to a paper posted to arXiv on June 17, 2026 [1]. The system, described as a hierarchical multi-modal retrieval framework, is designed to generate captions that include object attributes, event context, and underlying significance — details that traditional methods often miss [1]. It departs from earlier approaches by treating source articles as structured documents rather than monolithic blocks of text. The retrieval mechanism weighs textual components such as headlines and body sections, analyzes visual placement patterns, and computes similarities across content-to-visual, visual-to-visual, and discourse-positioning dimensions [1]. A contextual relevance refinement stage further filters the retrieved material before it is passed to the generation pipeline [1]. The pipeline operates in three stages. A vision-language model first produces a concise description of the image. Relevant segments are then extracted from the retrieved articles based on that description. Finally, a large language model combines the visual description with the extracted knowledge to produce a comprehensive, contextually detailed caption [1]. The source code has been publicly released on GitHub [2]. The framework was evaluated on the OpenEvent-V1 dataset, a collection built for the EVENTA challenge that tests captioning systems on news imagery [1]. On the private test set, the system achieved an overall score of 0.2824, securing fifth place in the competition [1]. Standardized benchmarks of this kind are maintained by academic and industry groups to track progress in language and vision tasks, using fixed datasets and evaluation metrics to compare model performance [4]. The work reflects a broader trend in machine learning toward retrieval-augmented generation, where models pull in external information rather than relying solely on parameters learned during training. The EVENTA challenge, held as part of the ACM Multimedia conference series, provides a venue for testing such systems on real-world news content. The paper’s authors argue that their hierarchical, multi-modal retrieval design offers a more nuanced way to ground captions in supporting evidence than previous single-source methods [1].