Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation

Researchers have introduced ELF-S2T, an audio-conditioned continuous-target generative model designed to handle both automatic speech recognition and speech-to-text translation within a single framework, achieving competitive results on standard benchmarks [1]. The model, detailed in a paper submitted to arXiv on June 9, 2026, is built upon a pre-trained Embedded Language Flows (ELF) backbone and processes speech through a frozen Whisper encoder followed by a single linear projector [1]. This architecture allows the system to prepend an audio condition to a noisy text latent, enabling in-context, flow-matching denoising for text generation [1]. Unlike conventional systems that generate discrete text tokens, ELF-S2T operates in a continuous representation space [2]. The authors note that the potential of continuous-target language modeling for speech-to-text tasks had remained unexplored prior to this work [2]. To address the risk of the model over-relying on its pre-trained text context, the team introduced a technique called audio forcing during training [1]. At inference, the audio condition is further amplified using classifier-free guidance [1]. Experiments conducted on the LibriSpeech and CoVoST2 datasets demonstrated competitive performance for both automatic speech recognition (ASR) and speech-to-text translation (S2TT) [1]. A key finding from the error analysis was that while ASR and S2TT errors appear distinct on the surface, both originate from the same underlying cause: a close distance confusion within the continuous latent space [2]. This observation suggests a common semantic mapping process underpinning both recognition and translation, aligning with the continuous representation generation paradigm [2]. The code and pretrained models have been made publicly available [1].