RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering

A new evaluation dataset called RECOM reveals a fundamental tradeoff in automatic metrics used to assess open-ended question answering, finding that no single metric excels at both measuring genuine content alignment and distinguishing between model performance levels. The RECOM dataset, introduced in a paper by Pushwitha Krishnappa, comprises 15,000 questions from r/AskReddit posted in September 2025, each paired with authentic community replies that postdate the training cutoff of every evaluated model [1]. Five open-source large language models (LLMs), ranging from 7 to 10 billion parameters, were scored against every reply in the dataset [1]. LLMs are a type of machine learning model designed for natural language processing tasks such as language generation, trained with self-supervised learning on vast amounts of text [6]. The evaluation exposed a clear tension. Cosine similarity effectively separated real human answers from random noise, achieving a Cohen's d of approximately 2, but it could not rank the five models, with an absolute d value below 0.1 [1]. BERTScore precision appeared to rank the models with a raw absolute d value up to 0.63, but this ranking collapsed to an absolute d of 0.09 after controlling for response length [1]. Its validity for separating real from random answers was also weaker, with a Cohen's d of roughly 0.8 compared to cosine similarity's 2 [1]. The paper argues this validity-discrimination tradeoff is a property of the metrics' representation design, not the models themselves, because every metric scored the same outputs [1]. Three independent LLM judges reproduced the validity gap and also separated the five models only weakly [1]. The authors recommend reporting metrics on both axes with an explicit random-baseline floor [1]. The work arrives as the evaluation of LLM outputs remains an active area of research and tooling. Platforms like Hugging Face have collaborated with arXiv to integrate interactive machine learning demos directly on paper abstract pages, allowing users to explore models without writing code [2][3]. These demos, built with open-source tools such as Gradio and Streamlit, aim to increase reproducibility and allow a wider audience to identify biases and other issues [2][4]. The RECOM dataset is publicly available, contributing a contamination-free benchmark to this broader effort of improving model assessment [1].