Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation

A systematic review of human evaluation protocols in computational linguistics has found pervasive under-reporting of study design details, raising concerns about the reproducibility of research on long-form text generation. The analysis, posted to arXiv on June 6, examined publications from *CL conferences spanning 2023 through 2025 [1]. Researchers manually reviewed 284 papers and used large language model-assisted methods to analyze more than 1,800 additional papers [1][2]. They defined a set of 20 reportable criteria tied to the reproducibility of human evaluation studies and applied those criteria to gauge current reporting norms [1][2]. The team found widespread under-reporting of important aspects of human evaluation study design, leading to ambiguity about what was measured and how, who contributed judgments, and how judgments should be interpreted [1][2]. Human evaluation remains a cornerstone for assessing the quality of generated text, but its reliability depends on transparent and well-documented protocols — details that are frequently missing in current practice [2]. The study’s authors note that without such documentation, it becomes difficult to replicate findings or compare results across experiments [1]. The paper does not single out individual conferences or institutions but instead describes a community-wide pattern of incomplete reporting [1][2]. The findings arrive as the broader field of artificial intelligence grapples with evaluation standards. Systems that generate long-form text — from conversational agents to digital companions — have expanded significantly with advances in large language models, affective computing, and social robotics [3]. These systems are designed to simulate companionship through social, emotional, or relational interaction, and their capabilities now include maintaining ongoing social engagement and cultivating interactions that resemble interpersonal relationships [3]. Reliable human evaluation is critical for measuring how well such systems perform outside of narrow task-oriented benchmarks [1][3]. Based on their findings, the researchers outline actionable recommendations to support more transparent and reproducible reporting in future research [1][2]. The analysis code and annotated dataset have been made available on GitHub through the Larch Lab repository [2]. The paper’s authors encourage the community to adopt the 20 criteria as a baseline for documenting human evaluation studies, with the goal of reducing ambiguity and strengthening the evidentiary value of human judgments in language-generation research [1][2].