Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings

Sequential preference training on language models does not cause uniform forgetting of earlier objectives, according to a study posted to arXiv on 18 June 2026. The work finds that outcomes depend on the relationship between objectives, signal strength, and training order [1][2]. The study examines Direct Preference Optimisation (DPO), a method for aligning models with multiple behavioural objectives applied one after another [1][2]. Researchers used Llama-3.1-8B-Instruct with LoRA adapters and evaluated all objectives after every stage against a fixed base-model reference [1][2]. They tested four preference settings: distributional conflict, multi-attribute interaction, a strong safety signal, and compatible response-quality objectives [1][2]. Results showed no single forgetting pattern. Preference change ranged from partial degradation to stability, pair-level redistribution, or positive transfer [1][2]. A pair-level analysis using length-normalised policy margins revealed that aggregate metrics can conceal heterogeneous changes across preference pairs. Quartile decomposition further showed that high-confidence pairs can either degrade or improve depending on the setting [2]. Mechanistic diagnostics provided additional detail. Stage 2 gradients and adapter updates were near-orthogonal to the previous objective across all settings, offering little evidence that direct gradient opposition drives the observed changes [1][2]. The authors suggest that future sequential alignment pipelines should account for objective compatibility and signal strength rather than assuming later objectives affect earlier preferences uniformly [2]. The paper was submitted to arXiv, an open-access repository of electronic preprints that, as of November 2024, receives about 24,000 articles per month [6]. arXiv hosts papers across mathematics, physics, computer science, and related fields, and its contents are moderated but not peer-reviewed [6]. The platform also supports community-built tools through arXivLabs, a framework launched in 2020 that enables collaborators to develop features such as citation explorers and code linkers directly on article pages [4][5].