IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages

A new multilingual benchmark called IndicContextEval has been released to test whether audio large language models genuinely use contextual prompts when transcribing speech across eight Indian languages, rather than relying solely on their pretrained knowledge [1]. The benchmark comprises 56 hours of natural speech from 555 speakers spanning 23 professional domains [1][2]. It employs a seven-level prompting framework that progressively introduces contextual signals. These range from metadata and natural-language domain descriptions to entity lists presented in both English and native scripts. The framework also includes adversarial prompts containing incorrect entities, designed to probe whether a model is truly attending to the provided context [1][2]. Five audio large language models were evaluated using IndicContextEval. The results revealed substantial differences in how the models utilised context, underscoring what the paper's authors describe as a critical need for explicit evaluation of contextual grounding in these systems [1][2]. The work addresses a gap in existing evaluation methods, which typically assess transcription under fixed prompting conditions and rarely incorporate explicit contextual inputs [2]. The paper was submitted to arXiv on 17 June 2026 under the Electrical Engineering and Systems Science category for Audio and Speech Processing [1]. It appears on the arXivLabs framework, which supports experimental projects developed with community collaborators who adhere to arXiv's values of openness, community, and user data privacy [1]. Audio large language models, or AudioLLMs, are designed to condition speech recognition on textual prompts such as domain descriptions or entity lists. The central question the benchmark investigates is whether performance improvements stem from genuine context utilisation or from parametric knowledge absorbed during pretraining [2]. The benchmark's multilingual design, covering eight Indian languages, aims to broaden the evaluation landscape beyond English-centric tests [1][2]. The release of IndicContextEval comes amid broader efforts to link research papers with practical artifacts. Platforms such as Hugging Face now support paper pages that aggregate models, datasets, and interactive demos associated with a given publication [4]. An integration between Hugging Face and arXiv also allows community-built demos to appear directly on a paper's arXiv abstract page, provided the Space's README file includes a link to the paper [5]. These infrastructure developments facilitate the kind of reproducible benchmarking that IndicContextEval represents.