UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation

Researchers have made advancements in machine translation with the development of UniTranslator, a unified multi-modal framework, and improvements in function vectors and low-resource language pairs.

UniTranslator is a unified multi-modal framework for end-to-end in-image machine translation that tightly couples translation understanding and text editing^[1]. It introduces an Understand-Generation Alignment Module (UGAM) to bridge the representation gap between understanding and generation, and a Spatial Mask Decoder (SMD) to improve spatial grounding and geometric alignment. UniTranslator has achieved state-of-the-art performance across diverse language directions and complex real-world layouts^[1]. Separately, researchers have investigated function vectors in machine translation for language-agnosticity, finding that translation FVs extracted from a single English$ o$X direction can transfer to other target languages^[2]. A machine translation system has also been developed for the low-resource Tangkhul-English language pair, with the primary ByT5-large system achieving a corpus BLEU score of 39.97 and a BERTScore F1 of 0.8104 on a held-out test set of 3,856 sentences^[3].