Model Unlearning Objectives Vary for Distinct Language Functions

Large language models acquire dangerous knowledge and generate toxic text during pretraining, and researchers now argue that removing these capabilities requires distinct unlearning methods tailored to each type of undesirable output, according to a new paper posted to arXiv [1]. The study, submitted on 26 May 2026, examines two mechanistically distinct unlearning goals: dangerous-knowledge unlearning and toxicity unlearning [1]. The authors contend that just as post-training uses different objectives to shape different behaviors, unlearning methods should be designed for the specific language function at issue [2]. For dangerous knowledge, the team introduced a cosine-based, meta-learned variant of RMU. For toxicity, they proposed a multi-layer objective based on layer-specific probe directions [2]. The methods were tested across four open-source models in the 7-8B parameter range [1]. The findings suggest that unlearning should be studied as a family of problems, analogous to the multiple types of LLM post-training [2]. This framing challenges a one-size-fits-all approach to model alignment and safety. The concept of unlearning in machine learning draws a loose parallel to organizational learning, where knowledge is created, retained, and transferred within an organization at individual, group, and organizational levels [4]. In the context of LLMs, however, the goal is the deliberate removal of specific knowledge or behavioral tendencies rather than its accumulation. The paper does not include direct quotations from the authors, but the abstract states the methods "achieve strong results, based on distinct training objectives for the two types of unlearning" [2]. The work was shared through arXivLabs, a framework that allows collaborators to develop and share new arXiv features directly on the platform [1]. The research contributes to ongoing efforts to make large language models safer by addressing the root causes of harmful outputs during the pretraining phase rather than relying solely on post-hoc filters or reinforcement learning from human feedback. While the paper focuses on English-language models, the broader challenge of managing undesirable model behaviors has implications for multilingual systems. Multilingualism in computing sits on a continuum between internationalization and localization, and software development nearly always uses English, with multilingual versions often produced as alternative options based on the English original [3]. As LLMs are deployed across languages, the question of whether unlearning methods transfer across linguistic contexts remains open.