FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision

A new unified video-to-audio framework called FoleyGenEx aims to close a persistent gap in synthetic sound generation by combining multi-modal control, frame-level temporal alignment, and fine-grained semantic precision in a single model, according to a preprint posted to arXiv on June 12, 2026 [1]. The framework, detailed in a paper submitted to the open-access repository, addresses a trade-off that has constrained existing video-to-audio methods: systems either offer multi-modal control with weak temporal alignment, or they achieve strong alignment but lack reference audio conditioning and semantic detail [1][2]. FoleyGenEx introduces three core mechanisms to bridge that divide. A conditional injection mechanism handles audio-controlled video-to-audio generation and Foley extension, while a multi-modal dynamic masking strategy preserves synchronization during training [1][2]. An adverb-based data augmentation algorithm, which draws on signal processing and large language models, strengthens textual supervision by injecting nuanced semantic information [1][2]. Large language models are machine learning systems trained on vast text corpora for natural language tasks [8]. The authors evaluated FoleyGenEx on three established benchmarks: AudioCaps, VGGSound, and Greatest Hits [1][2]. The preprint reports competitive controllable video-to-audio performance against existing methods, though specific quantitative metrics were not detailed in the abstract [1][2]. Demo samples have been made available on a dedicated project page [1][2]. The paper appears on arXiv, a repository that hosts electronic preprints across physics, mathematics, computer science, and related fields [6]. As of November 2024, the platform receives roughly 24,000 new articles per month and has surpassed two million total submissions since its launch in 1991 [6]. Submissions are moderated but not peer-reviewed before posting [6]. The FoleyGenEx preprint is accompanied by a suite of experimental community tools developed under arXivLabs, a framework that allows third-party collaborators to build features directly on the article record page [4][5]. These tools include the Bibliographic Explorer for citation-tree navigation, the CORE Recommender for discovering related open-access papers, and Connected Papers for visualizing relevant literature [4][5]. arXivLabs collaborators operate under guidelines that require adherence to openness, community, excellence, and user-data privacy, with access limited to minimal and anonymized data [4].