Conversational Domain Adaptation of IndicTrans2 across 21 Indic Languages via Experience Replay and Model Soups

Researchers have adapted the IndicTrans2 machine translation system to handle casual, conversational English-to-Indic text across 21 languages, using two established techniques to preserve general translation quality while significantly improving performance on informal input. The work targets a known limitation of IndicTrans2, which is described as the strongest open English-to-Indic translation system but one that tends to sound stiff on conversational input because it is trained on general text [1]. The team, which includes Aditya Pratap Singh, fine-tuned the 1-billion-parameter variant of the model using only public datasets: OpenSubtitles, BPCC-H-Daily, and Tatoeba [1]. Plain fine-tuning on this conversational data improved the chrF metric but caused the model to forget its general-domain knowledge, with a drop of 3.9 chrF on the FLORES benchmark for Hindi [1]. To eliminate this trade-off, the researchers mixed general training data back into the fine-tuning process — a method known as experience replay — and then averaged the weights of the fine-tuned model with the original base model, a technique called model souping [1]. The resulting model outperforms the base IndicTrans2-1B on conversational chrF in every one of the 21 languages, with a mean gain of 6.2 [1]. At the same time, performance on the general-domain FLORES benchmark remained essentially unchanged, with a mean change of -0.17 and all language-level changes falling within 0.7 chrF [1]. Paired bootstrap tests confirmed the conversational gains are statistically significant, with a p-value of 0.004 or less, and that FLORES performance is not significantly degraded [1]. The authors are explicit about the limits of their findings. A blind human evaluation and a check using multiple large language models did not confirm the chrF gains as a perceived improvement in translation quality. The paper therefore treats the conversational gain as largely a register match to the reference translations rather than proof of better translation [1]. The techniques themselves are not novel; the contribution is an end-to-end study in the Indic conversational setting [1]. The conversational evaluation data comes from IN22-Conv, the conversation-domain subset of the IN-22 benchmark, which contains 1,503 sentences translated across 22 Indic languages and is designed to assess translation quality in day-to-day conversational applications [7]. The adapted model has been released on Hugging Face, and the base IndicTrans2 model card notes that the system was originally detailed in a 2023 paper published in Transactions on Machine Learning Research [5][6].