On the Limits of Model Merging for Multilinguality in Pre-Training

Merging language models that were pre-trained on a single language leads to a severe drop in performance, according to a new controlled study. Researchers found that interference between the models causes a collapse, challenging assumptions about the flexibility of model merging techniques. The study, submitted on 25 May 2026, tested whether post-training model merging could be applied to models that were pre-trained monolingually [1]. The authors conducted a controlled comparison of mixed, merged, and monolingual pre-training setups [2]. While monolingual pre-training resulted in strong performance within a single language, any attempt to merge these monolingual models caused a performance collapse [1]. The researchers attributed this failure to interference between the models [2]. Their analysis indicates that representational similarity is a prerequisite for successful model merging [1]. The findings lead to the conclusion that the flexibility observed when merging models during fine-tuning does not trivially extend to the language-specific pre-training phase [2]. This work provides a direct empirical boundary for a technique that has gained traction as an alternative to mixing pre-training data for achieving multilingual capabilities [2]. The study is hosted on arXiv, a repository for scientific papers, and is associated with arXivLabs, a framework for community-developed features [1]. The broader field of natural language generation, which encompasses the development of such models, has evolved significantly since the 1960s, with modern systems often relying on statistical models trained on large corpora of human-written text [4].