Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis

A research team has formalized a multimodal extension of the Shapley Value framework to explain how models integrating text and audio arrive at decisions, according to a paper submitted on 21 Apr 2026 [1][2]. Multimodal Large Language Models (MLLMs) combine text and audio to interpret context in interactive dialogues, but the internal influence of each modality has remained opaque [1][2]. Shapley Values, a model-agnostic explainability tool widely used in text-based natural language processing, have been difficult to apply to multimodal data because of cross-channel dependencies, complex dialogue structures, and the computational cost of dense audio representations [1][2]. The new work treats discrete text tokens and aligned audio segments as cooperative features within the Shapley framework [1][2]. To manage computational demands, the authors deploy exact Shapley Value computation for low-dimensional inputs and sampling-based approximations, including Monte Carlo permutations and stratified sampling with Neyman-optimal allocation, to reduce variance under limited computational budgets [1][2]. A preprocessing method called Spectrogram-Guided Phonetic Alignment (SGPA) maps high-frequency audio streams to interpretable, word-aligned segments, resolving the granularity mismatch between modalities [1][2]. The researchers provide an open-source, model-agnostic Python package and a companion graphical user interface for computing and interactively visualizing multimodal attributions [1][2]. They evaluated the framework using curated subsets of the VoiceBench and Infinity Instruct datasets across multilingual scenarios [1][2]. The experiments showed that input modality is a primary driver of attribution volatility and that standard syntactic importance proxies often fail to predict model attention in multimodal, cross-lingual contexts [1][2]. The paper appears on arXiv under the Computation and Language category [1].