AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems

A new benchmark called AgentLeak reveals that multi-agent large language model systems leak sensitive data through internal communication channels at rates far higher than what final-output audits detect, according to research posted on arXiv [1]. The study, authored by Faouzi El Yagoubi and colleagues, introduces AgentLeak as a tool for evaluating internal-channel privacy leakage in multi-agent LLM systems [1]. Large language models are neural networks trained on vast text corpora for tasks such as generation, summarization, and analysis, and they underpin modern chatbots [3]. When multiple LLM agents coordinate on a task, sensitive information can pass through inter-agent messages, shared memory, and tool arguments — pathways that standard output-level benchmarks do not inspect [1]. AgentLeak instruments seven privacy-relevant communication pathways and was tested across 1,000 scenarios spanning healthcare, finance, legal, and corporate domains [1]. The evaluation used five production LLMs — GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Mistral Large, and Llama 3.3 70B — and generated 4,979 validated execution traces [1]. The findings show that multi-agent configurations reduce final-output leakage to 27.2 percent, compared with 43.2 percent in single-agent mode [1]. However, the introduction of internal coordination channels raises total system exposure to 68.9 percent when aggregating final outputs, inter-agent messages, and shared memory [1]. Inter-agent messages alone leak at a rate of 68.8 percent, meaning output-only audits miss 41.7 percent of violations [1]. The pattern in which inter-agent leakage equals or exceeds final-output leakage held consistently across all five models and all four domains tested [1]. The research suggests that privacy risk in multi-agent systems is shaped more by architectural coordination channels than by final-output behavior alone, and that these internal channels remain invisible to standard output-level defenses [1]. The benchmark focuses on a coordinator-worker setting, and the authors note that the results may not generalize to all multi-agent architectures without further study [1].