NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translation

Researchers have made advancements in speech-to-speech translation, introducing frameworks to minimize latency and improve translation quality. Two recent studies address the challenges of simultaneous translation and error reduction.

Simultaneous speech-to-speech translation aims to enable near-real-time communication by minimizing latency^[1]. However, excessive pursuit of low latency can result in fragmented chunk-wise speech with frequent pauses^[1]. To address this, the NaturalFlow framework was introduced, minimizing inter-chunk silences by leveraging model-internal signals^[1]. Experiments show that NaturalFlow produces natural speech flow while maintaining competitive latency and translation quality^[1]. Meanwhile, another study addressed error propagation in cascaded speech translation systems, which suffer from incorrect Automatic Speech Recognition (ASR) outputs^[2]. The study found that most ASR substitution errors in Vietnamese speech translation arise from phonetic confusions rather than random noise^[2]. A method called PiDA was proposed, which improves translation of erroneous ASR outputs by up to +2.04 BLEU over standard fine-tuning^[2]. PiDA uses phonetic word embeddings to generate ASR-like corruptions^[2].