Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy

A new metric called READ evaluates automatic speech recognition hypotheses directly from the speech signal, eliminating the need for reference transcriptions. The approach, described in a paper submitted to arXiv on 3 June 2026, uses a pretrained text-to-speech model to measure acoustic discrepancy between speech and text [1]. Traditional evaluation of automatic speech recognition (ASR) systems relies heavily on reference-based metrics such as Word Error Rate (WER), which require labor-intensive human-labeled transcripts [5]. Reference-free alternatives have typically depended on internal confidence scores from ASR decoders or auxiliary language models, but confidence scores often suffer from poor calibration and overconfidence, while text-only approaches overlook the speech signal itself [5]. READ, which stands for Reference-free Hypothesis Evaluation with Acoustic Discrepancy, addresses these limitations by restoring emphasis on the acoustic model component of ASR [5]. The metric uses a pretrained auto-regressive TTS model to compute the conditional likelihood of speech tokens given a text hypothesis, producing a fine-grained measure of acoustic discrepancy between speech and text [1]. Without any additional training, READ can be applied for hypothesis refinement [1]. Experiments show that READ correlates with specific recognition errors and improves ASR outputs, achieving up to 20% relative error rate reduction, with particularly strong gains under noisy conditions [1][2]. The work draws on a broader trend in speech processing toward reference-free evaluation. A separate 2023 paper introduced NoRefER, a referenceless quality metric that fine-tunes a multilingual language model using contrastive learning and a Siamese network architecture to rank ASR hypotheses by quality without ground-truth transcripts [4]. That method exploits known quality relationships between hypotheses generated from multiple compression levels of an ASR model for self-supervised learning [4]. Another line of research, described in a 2026 study on pathological speech, proposed an explainable ASR Inconsistency Score that uses two distinct ASR models to estimate intelligibility without manual transcriptions, achieving high correlation with expert perceptual ratings across Dutch, Spanish, and English [3]. The READ authors argue that while language model rescoring reinforces linguistic plausibility, a corresponding mechanism for acoustic grounding has been notably absent from reference-free evaluation pipelines [5]. By computing the conditional likelihood of speech tokens given a text hypothesis, READ provides a direct acoustic verification that text-only and confidence-based methods cannot offer [1][5].