Output Vector Editing for Memorization Mitigation in Large Language Models

Researchers have proposed a new method called output vector editing to reduce the tendency of large language models to memorize and reproduce verbatim training data, a behavior that poses privacy and copyright risks [1]. The technique, detailed in a paper by Ahmad Dawar Hakimi and colleagues, targets the output vectors of multi-layer perceptron (MLP) neurons rather than simply zeroing out their activations, which has been the conventional approach [1]. The activation only controls whether a neuron engages; the output vector is what writes information into the model's residual stream and, through superposition, can encode multiple features [2]. By minimally modifying these output vectors, the method introduces a distractor in the model's vocabulary space, redirecting the residual-stream contribution while leaving the neuron's activation unchanged [2]. The researchers evaluated the approach on four models ranging from 360 million to 7 billion parameters, including SmolLM-360M, OLMo-1B, OLMo-7B, and Llama2-7B [2]. They centered their analysis on OLMo-7B, whose open weights and pretraining corpus allowed for systematic mining of 6,831 memorized sequences [1]. The output vector editing method achieved up to 87.9% suppression of these memorized sequences [1]. A 2.7× gap in performance over simply zeroing out the same located neurons confirmed that the suppression stems from the output-vector edit itself, not from the neuron localization step alone [2]. The study defines four edit modes that range from aggressive suppression to minimal redirection [1]. When used as an ensemble, these modes cover 96.5% of memorized sequences, while a recommended single-mode configuration reaches 81.5% suppression without causing catastrophic failures in the model's general performance on other tasks [1]. The work also identified a mechanistic boundary: approximately 14% of memorized sequences are unreachable through MLP-only editing [2]. For these failures, the researchers found that ablating the top contributing attention heads recovers 60–64% of the sequences, with stronger recovery on continuations that copy tokens directly from the input prefix [2]. This positions the attention mechanism as a complementary fallback rather than the primary driver of memorization [1]. The authors report that the ordering of edit modes and the trade-off between success and locality transfer across all four tested models, with success rates scaling with model size rather than the specific model family [2].