Curvature-Guided LoRA: Matching Full Fine-Tuning in Function Space

Researchers have proposed two new methods, Curvature-Guided LoRA and Sparse Memory Finetuning, to improve the adaptation of large pretrained models to new tasks.

Curvature-Guided LoRA, proposed in a paper submitted to arXiv on March 31, 2026, and revised on June 7, 2026^[1], uses local curvature information to select adaptation directions, matching full fine-tuning in function space. The algorithm improves performance and convergence speed of parameter-efficient fine-tuning methods like LoRA. LoRA enables efficient adaptation of large pretrained models, but often lags behind full fine-tuning in convergence speed and final performance^[1]. Meanwhile, a separate paper on arXiv proposes Sparse Memory Finetuning (SMF), a low-forgetting alternative to LoRA and full finetuning. SMF adds key-value memory layers to the model and updates only the small set of memory rows that the current batch reads most heavily, improving MedMCQA by 2.5 percentage points while keeping forgetting probes within roughly 1 point of the base model^[2]. In contrast, LoRA and full finetuning achieve larger gains but with clear drift on both forgetting probes^[2].