When Should Memory Stay Silent: Measuring Memory-Use Boundaries in Memory-Augmented Conversational Agents

Researchers have introduced RBI-Eval, a study to measure when memory should be integrated into responses in memory-augmented conversational agents, revealing substantial behavioral divergence between models with and without access to sensitive memory.

The RBI-Eval study compares model behavior with and without access to sensitive memory under identical benign prompts, evaluating four base LLMs against a matched no-memory reference across four memory-access settings^[1]. Findings show that with memory available, the separation score for sensitive-memory integration decreases by 8.9%--26.6% relative to the matched no-memory reference for GPT-5.4-mini, and by 51.1%--82.9% for Claude-Sonnet-4.6, DeepSeek-V4-Flash, and Qwen3.5-9B. Control experiments on DeepSeek and GPT-5.4-mini indicate this effect is specific to sensitive content. Separately, a study on Search-Time Contamination (STC) found that it can inflate performance by up to 4%^[2]. This suggests that existing evaluations may overestimate true reasoning ability due to STC. The RBI-Eval findings imply that safe personalization requires memory-aware decisions at both retrieval and generation time.