Personalized Turn-Level User Conversation Satisfaction Benchmark

A new benchmark and evaluation tool aim to measure personalized user satisfaction with AI assistants, addressing a gap in how conversational models are assessed. The system, detailed in a paper submitted 28 May 2026, combines user memory with conversation context to judge whether a response meets an individual's expectations at a specific turn [1]. The evaluator produces satisfaction scores and rationales focused on dissatisfaction, moving beyond generic quality metrics that cannot capture why the same response might please one user and frustrate another [2]. Meta-evaluation against human annotations showed that incorporating personalized memory and applying post-hoc score calibration improved ordinal agreement and the detection of dissatisfied turns compared to supervised, retrieval-based, and generic large language model baselines [1][2]. The researchers also introduced PersTurnBench, a benchmark that uses the verified evaluator to assess generation models through a replay mechanism. By holding the replay state fixed, PersTurnBench allows a controlled comparison of generic generation models against memory-augmented personalized systems without requiring new human labels for each candidate model [1][2]. This approach lets researchers compare models on personalized satisfaction without collecting fresh user feedback for every iteration [2]. The work is situated within the broader study of customer experience, which encompasses the cognitive, emotional, sensory, and behavioral responses of a person during all stages of interaction with a product or service, including pre-purchase, consumption, and post-purchase phases [4]. The remembered experience, a component of customer experience, relates to a recollection of memories about previous events and experiences with a product or service [4]. The new evaluator operationalizes a similar concept by using compact user memories to inform satisfaction judgments at each turn [2]. The development of such evaluation tools comes as AI assistants, particularly large language models, have seen rapid deployment. OpenAI's release of ChatGPT in November 2022 catalyzed widespread interest in generative AI, and the organization has since developed the GPT family of models that underpin many conversational systems [5]. As these assistants become more integrated into daily digital life, the ability to measure whether they satisfy individual users over time grows more critical. The PersTurnBench framework offers a structured method for that measurement, potentially guiding the development of more personally attuned AI systems [1][2].