Erased, but Not Gone: Output Forgetting Is Not True Forgetting

Standard evaluations of machine unlearning may systematically overstate how thoroughly models forget, according to a study submitted to arXiv on 23 Jun 2026 [1]. Researchers argue that output-level forgetting metrics can coexist with structured, retraining-inconsistent residuals hidden in a model’s representation space [1]. The paper, titled “Erased, but Not Gone: Output Forgetting Is Not True Forgetting,” challenges the common practice of judging machine unlearning by surface-level metrics such as low forget-set accuracy or reduced logit-level membership inference [1]. The authors instead propose using a retrained model — one trained from scratch without the data to be forgotten — as an operational reference for correct forgetting [1]. Across multiple unlearning methods, datasets, and model architectures, their empirical results show that standard output-level evaluation can systematically overestimate the success of unlearning [1]. Under this stricter lens, current methods often appear to have forgotten data at the output layer while exhibiting a structured mismatch relative to the retrained baseline [1]. The mismatch is not random noise; it is characterized by forget/retain asymmetry, directional mismatch, and concentrated residuals along retraining-related directions [1]. The models partially align with retraining on forget samples but remain more inconsistent on retain samples, leaving a residual discrepancy that output-level checks fail to capture [1]. The study’s findings arrive as machine unlearning gains attention as a potential tool for enforcing data privacy rights and complying with regulations that require deletion of user information. Large language models, which are trained on vast corpora of text through self-supervised learning, present a particular challenge for verifiable forgetting given their scale and the opacity of their internal representations [11]. The arXiv preprint server, where the study was posted, has served as a primary distribution channel for such computer science research since its founding in 1991, and now receives approximately 24,000 new articles per month [9]. The authors conclude that current machine unlearning is frequently evaluated for apparent forgetting rather than retraining-consistent forgetting [1]. Their work suggests that without representation-level audits, practitioners may be certifying the erasure of data that, in a structural sense, is not fully gone [1].