Robustness assessment of large audio language models in multiple-choice evaluation

A new study finds that large audio language models produce substantially different results on multiple-choice benchmarks when the order of answer choices is shifted or questions are paraphrased, challenging the reliability of standard evaluation practices that report a single accuracy figure [1]. The research, conducted by Fernando López and submitted to arXiv, systematically tested four models — Audio Flamingo 2, Audio Flamingo 3, Qwen2.5-Omni-7B-Instruct, and Kimi-Audio-7B-Instruct — across three benchmarks: MMAU, MMAR, and MMSU [1][2]. The authors found that existing multiple-choice question answering (MCQA) frameworks do not account for this variability, instead reporting one accuracy number per benchmark or category [1]. Language model benchmarks are standardized tests designed to evaluate performance on tasks such as language understanding, generation, and reasoning [3]. They typically consist of a dataset and corresponding evaluation metrics, with accuracy being a primary measure [3]. However, the study indicates that subtle perturbations in test design can cause significant swings in model scores, undermining the assumption that a single accuracy figure reliably captures a model's capability [1][2]. The sensitivity observed in large audio language models mirrors broader concerns about algorithmic bias in machine learning systems. Bias can emerge from how data is coded, collected, or selected for training, and can lead to systematic and unfair discrimination [5]. In the context of evaluation, the choice of question phrasing and answer ordering can privilege certain outcomes, a phenomenon that the proposed protocol seeks to mitigate [1][5]. Large language models, the broader family to which these audio models belong, are neural networks trained on vast amounts of data for natural language processing tasks [6]. Foundation models, a related concept, are trained on massive datasets so they can be applied across a wide range of use cases, including audio, images, and text [8]. The robustness of their evaluation is critical as these systems are deployed in high-stakes domains. To address the identified shortcomings, the authors propose a simpler evaluation protocol and metric that account for subtle variations in question and choice presentation [1][2]. The new approach aims to provide a more detailed evaluation report, moving beyond a single accuracy number to capture the stability of model performance under different test conditions [1]. The submission, which spans two versions totaling 99 KB and 93 KB respectively, was last revised in June 2026 [1].