Curvature-Guided Mixing for MLLM Adaptation

A new framework called Curvature-Guided Mixing (CGM) aims to solve a persistent problem in artificial intelligence: fine-tuning multimodal large language models for specialized tasks without erasing their general knowledge, a phenomenon known as catastrophic forgetting [1]. The method was detailed in a paper submitted to arXiv on June 23, 2026 [1]. When MLLMs are adapted for a specific job, they often lose the broad capabilities they were originally trained on. Existing techniques to merge a fine-tuned model with its pre-trained version are frequently heuristic or rely on sub-optimal objectives, the researchers state [1]. CGM takes a different, mathematically grounded approach. It formulates a joint optimization objective and uses a second-order, or Hessian, approximation of the loss landscapes to analytically derive an optimal, closed-form “soft mixing” ratio [1]. This ratio blends parameters based on their relative task-specific curvatures, effectively measuring how sensitive each parameter is to changes in the task [3]. The team also introduced a variant called CGM†, a “hard mixing” strategy that reframes the problem as a sparse parameter selection task [1]. This method starts from the fine-tuned model and treats the pre-trained parameters as a “knowledge reservoir” to be sparsely re-integrated [3]. For each parameter, a discrete decision is made on whether to revert to the pre-trained value, guided by a curvature-aware score [3]. By selecting only the top K% of parameters most critical for general knowledge and least critical for the new task, CGM† performs a targeted reversion that preserves foundational knowledge while maintaining the newly acquired skill [3]. Experiments were conducted on the LLaVA-1.5 and Qwen2.5VL model architectures across multiple downstream tasks [1]. The results showed that both CGM and CGM† consistently improved the trade-off between task specialization and general knowledge retention when compared to existing methods [1]. The code for the project has been made publicly available on GitHub [1]. The paper appears on arXiv, an open-access repository for electronic preprints that, as of late 2024, receives about 24,000 submissions per month and is not peer-reviewed [9].