From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models

A new study reveals that split large language models are vulnerable to data leakage from both user prompts and the model's own generated responses, expanding the known privacy risks of the distributed learning technique. The research, submitted to arXiv on 12 June 2026, identifies a novel attack method that can extract private information from both sides of a split-LLM interaction [1]. Split learning has gained traction as a way to fine-tune and run large language models on devices with limited computational resources, but it introduces specific privacy weaknesses [2]. Prior investigations focused almost exclusively on the leakage of private input prompts through inversion attacks on intermediate data representations [2]. The new work finds that the model's generative output responses represent an additional, largely unexplored, channel for sensitive information exposure [2]. The attack, named Patched Model Inversion with Dual-Sided Initialization (PIDI), operates in two stages to simultaneously target private input prompts and output responses [2]. It employs a dual-sided initialization approach combined with a patched inversion strategy designed to handle long sequences, which the authors state substantially outperforms earlier inversion methods [2]. To counter the threat, the researchers also propose a defense mechanism called the Adapter-based DualGuard with Mutual Information Defense (ADMI) [2]. This defense integrates an adapter-based local warmup strategy with mutual information regularization, aiming to provide strong empirical privacy protection while minimizing any impact on the model's task performance [2]. The paper reports that extensive experiments across diverse tasks and models show ADMI effectively defends against PIDI and other state-of-the-art inversion attacks [2]. The code for the project has been made publicly available on GitHub [2]. The findings arrive as large language models are increasingly deployed in privacy-sensitive domains, where users must weigh the risk of data exposure through external application programming interfaces against the high cost of running models locally [2]. The research was conducted under the arXivLabs framework, a platform that allows collaborators to develop and share new features while adhering to arXiv's values of openness, community, excellence, and user data privacy [1].