Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Models

Researchers have proposed new methods to improve multimodal models, addressing issues such as hallucination in video understanding and enhancing representation.

A new method, ViSSRes, has been introduced to mitigate hallucinations in Video Large Multimodal Models. ViSSRes is a lightweight MLP-style network that enhances video representations, reducing the hallucination rate of LLaVA-NeXT-Video by 40.69% and improving video understanding on MMVU by 18.36% ^[1]. Additionally, a learnable vision-only retrieval framework called GRIP has been developed, which learns to distinguish beneficial from detrimental in-context examples through contrastive training. GRIP improves consistently over similarity-based retrieval on three multimodal tasks ^[2]. Furthermore, a new approach called GIANT has been proposed to navigate gigapixel pathology images using large multimodal models. GIANT outperforms models specialized for pathology question answering on four out of five benchmarks and introduces a new benchmark suite, MultiPathQA, which includes 934 questions over 868 unique Whole Slide Images (WSIs) and five clinical challenges ^[3].