Aligning Audio Captions with Human Preferences

A new audio captioning framework uses human feedback instead of ground-truth captions to train models that produce descriptions listeners prefer, according to a preprint posted to arXiv and last revised on 23 June 2026 [1]. The approach, detailed by researchers led by K. Ashmic Hegde, replaces the standard supervised learning pipeline with Reinforcement Learning from Human Feedback (RLHF) [1]. Standard audio captioning systems rely on paired audio-caption datasets, which are expensive to create and often fail to capture what human listeners actually value in a description [1]. Captions, broadly defined, are texts representing audio content and can include written descriptions of music or sound effects, a feature especially important for deaf or hard-of-hearing audiences [5]. The team built a reward model on top of a Contrastive Language-Audio Pretraining (CLAP) embedding, training it on pairwise human preference judgments [3]. That reward model then guides an RL fine-tuning step that can be applied to any baseline captioning system without requiring additional labeled audio-caption pairs [4]. The authors report that their custom reward model achieved a weighted deviation of 4.19, the lowest among all automatic metrics tested, indicating the closest alignment with human evaluations [3]. In human evaluations across multiple datasets, captions produced by the RLHF-tuned system were preferred over those from baseline models, especially in cases where the baseline generated incorrect or unnatural descriptions [1]. The framework also matched the performance of fully supervised approaches that had access to ground-truth data, while using none itself [1]. The authors note that collecting pairwise preference data is significantly easier and less resource-intensive than obtaining ground-truth captions, which makes the method more scalable [4]. The work lands amid broader efforts to align machine-generated descriptions of audio with human judgment. A separate 2024 study proposed AlignCap, a framework for speech emotion captioning that uses preference optimization to reduce hallucinations and improve consistency on unseen speech [9]. That system combined knowledge-distillation regularization with preference optimization to align large language model outputs with human expectations in zero-shot settings [10]. The new RLHF-based audio captioning framework extends the preference-alignment concept to general audio scenes, with the authors suggesting that further gains are possible through expanded preference data and advanced RL techniques [3].