Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMs

A new study posted to arXiv demonstrates that an adversary can extract unseen training data from large language models by poisoning a fraction of the training dataset, reshaping the model’s internal loss landscape to force memorization of a targeted record [1][2]. The paper, authored by Md Abdullah Al Mamun and submitted on June 15, 2026, introduces an attack that requires no architectural changes to the model and works across both centralized and federated learning setups [1][2]. The core mechanism involves injecting poisoned data that creates a sharp loss minimum at a specific target completion, while raising loss on nearby alternatives. This manipulation compels the model to treat the target as the only low-loss solution in its neighborhood, leading to memorization [2]. The researchers report that the technique amplifies privacy leakage, achieving up to 100% successful extraction for language models and up to 90% for vision-language models [1][2]. Large language models, which are machine learning systems with many parameters trained on vast text corpora through self-supervised learning, are increasingly fed proprietary or sensitive information, including private healthcare records, financial data, and user conversations containing secrets [2][8]. The study frames this trend as a central privacy concern, noting that the attack is effective even when the adversary has no direct access to the target record [2]. The authors found that training a model with differential privacy successfully thwarted the initial poisoning method. However, they also developed a secondary attack that directly probes the loss landscape, bypassing those differential privacy defenses [1][2]. The paper’s submission file is listed at 18,533 KB [1]. arXiv, where the paper appears, is an open-access repository for electronic preprints that has been operating since 1991 and now receives about 24,000 submissions per month [6]. Papers on the platform are moderated but not peer-reviewed [6]. The repository also hosts arXivLabs, a framework for community-developed tools that appear on article pages, though the Labs program is currently pausing new proposals while the development team focuses on migrating systems to the cloud [3][4].