MosaicLeaks: Can your research agent keep a secret?

A new benchmark called MosaicLeaks reveals that deep-research AI agents routinely leak sensitive enterprise information through their web queries, and that simply instructing them to be careful does little to stop the problem [1]. The benchmark, which contains 1,001 multi-hop research chains, is designed to test how agents handle tasks that require interleaving private local documents with public web searches [1]. The core risk is the "mosaic effect," where an adversary observing an agent's outbound query log can piece together fragments of private data, even if no single query reveals a complete secret [1]. The dataset is split into 559 training chains, 98 validation chains, and 344 held-out-company test chains [1]. Researchers found that a base Qwen3-4B model achieved a strict chain success rate of 48.7% while exhibiting an answer or full-information leakage rate of 34.0% [1]. When the model was trained solely for task performance, its success rate climbed to 59.3%, but leakage worsened significantly to 51.7% [1]. This illustrates a central tension: more informative web queries improve task accuracy but also provide more fragments for an observer to exploit [1]. The study also tested a simple prompt-based defense, telling the agent not to issue web queries that leak local information. For the Qwen3-4B model, this approach reduced leakage from 34.0% to 25.5%, but it also caused strict chain success to drop from 48.7% to 44.5% [1]. The primary behavioral change was a reduction in the number of web queries, not consistently safer query construction [1]. To address this, the team developed a training method called Privacy-Aware Deep Research (PA-DR), which combines a situational task reward with a learned privacy reward [1]. The privacy component uses a classifier to estimate whether new web queries leak information directly or create a new mosaic leak when added to the existing query log [1]. After training with PA-DR, the model's strict chain success rose to 58.7% while its leakage rate plummeted to 9.9%, a figure lower than the untrained base model [1]. The agent achieved this not by searching less, but by dropping revealing details such as specific metrics and dates from its queries [1]. The broader AI landscape has seen a surge in the deployment of large language models by major firms. Microsoft, one of the companies referenced in the benchmark's example chains, has heavily invested in AI and cloud computing through its Azure platform [10]. Meanwhile, companies like Anthropic have centered their corporate identity on AI safety, developing models such as Claude with techniques like "constitutional AI" to improve ethical compliance [8][9]. The MosaicLeaks research suggests that for agentic systems, safety must be actively trained into the search behavior itself, rather than relying on pre-existing model alignment or simple instructions [1].