What sentiment analysis can't see: Measuring whether customers were helped, and what went wrong, across 70,000 support conversations

A study of 70,450 customer support conversations found that using a large language model to estimate satisfaction correlated with customer ratings at 0.47, outperforming traditional sentiment analysis, which correlated at 0.36 [1][2]. The research, posted on arXiv, used GPT-5.4 to read conversations from an online fundraising platform and estimate whether a customer was satisfied, while also flagging if a concrete problem was reported [1][2]. These estimates were validated against the 1-to-5 star ratings customers left after their interactions [2]. The satisfaction estimate not only tracked ratings more closely but also identified unhappy customers with fewer false alarms than sentiment analysis [2]. A key finding was the frequent divergence between a customer's tone and their actual satisfaction. The study reported that tone and satisfaction disagreed in 44% of conversations [1][2]. A single "Neutral" sentiment label was found to obscure a wide range of customer states, from those who were quietly satisfied to those who had silently given up on a resolution [2]. The largest identified customer segment was labeled "tolerated friction" [1][2]. These customers expressed satisfaction but simultaneously reported a fixable problem, a cohort that sentiment-based dashboards cannot surface [2]. The authors argue this demonstrates that large language model annotation can capture a customer's state and the cause of their issue directly from raw text, moving beyond the tonality of language [2]. The paper's broader conclusion suggests a potential shift in how businesses derive metrics from support interactions, grounding them in customer outcomes rather than linguistic tone [2]. The code and data associated with the article were made available on Hugging Face [1].