ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

A team of researchers has introduced ORCA, a model-based method for scoring open-ended answers from large audio language models, according to a preprint posted on arXiv. The approach combines human annotation with machine correction to improve reliability over existing automated judges. The system, detailed in a paper by Santosh Kesiraju and colleagues, targets a growing problem in audio AI evaluation: as benchmarks add complex reasoning and subjective tasks, they demand free-form responses that are harder to grade consistently [1][2]. ORCA addresses this through a three-stage annotation pipeline that layers human judgment, structured feedback, and human-AI correction [2]. The process produced 9,663 annotations across 3,699 question-answer pairs drawn from 15 large audio language models on three audio understanding and reasoning benchmarks [2]. Inter-annotator agreement reached a Krippendorff's alpha of 0.82 [2]. Using curriculum learning, the ORCA models achieved a Spearman correlation of 0.91 with average human correctness ratings on benchmarks seen during training [2]. On previously unseen benchmarks, the correlation held at 0.85, outperforming several large language model judge baselines, including Gemini 2.5 Flash [2]. The paper also shows that ORCA's predicted variance aligns with human disagreement, enabling it to flag problematic benchmark items [2]. The work appears on arXiv, an open-access repository that hosts preprints across physics, computer science, and related fields without peer review [9]. The platform, which began in 1991, surpassed two million articles by the end of 2021 and now receives roughly 24,000 submissions per month [9]. The ORCA paper is listed under the Sound (cs.SD) category and was submitted in November 2025, with a revised version posted in June 2026 [1]. arXiv also hosts community-built tools through its arXivLabs framework, which allows third-party developers to add features such as citation explorers and code finders to article pages [8][7]. The framework, launched in 2020, requires partners to follow arXiv's values of openness, community, excellence, and user data privacy [8]. The ORCA abstract page includes links to several of these tools, including Bibliographic Explorer and CORE Recommender [1][7].