Meaning in Order, Order in Meaning: Semantic R-precision for Keyphrase Evaluation

A new evaluation metric called Semantic R-Precision (SemR-p) aims to close a gap in how automatically generated keyphrases are judged, by rewarding semantically relevant predictions that appear early in a ranked list rather than relying solely on exact word matches [1]. The metric, detailed in a paper submitted to arXiv on 5 June 2026, integrates semantic similarity into the established R-Precision framework, which evaluates only the top-R predictions where R equals the number of reference keyphrases [1][3]. Traditional keyphrase evaluation has long depended on exact lexical matching, an approach that fails to recognize predictions that are conceptually correct but phrased differently [1][2]. Other metrics have incorporated semantic similarity but ignored the order in which predictions are presented, a factor the authors argue is critical because human readers process keyphrases as holistic concepts and pay more attention to items appearing first [3][4]. SemR-p addresses this by using a two-part scoring logic. If a predicted keyphrase has an exact stem match with a reference, it receives a full score. For non-matches, the metric calculates a semantic score by averaging the cosine similarity between phrase-level embeddings of the prediction and the top-k most similar reference keyphrases [3][4]. This design is intended to reduce noise from unrelated concepts and mimic how humans focus on salient information [3]. The metric then averages these scores across the top-R predictions to produce a single document-level figure [4]. The work builds on a broader push within the field toward semantic-based evaluation. In 2024, researchers introduced KPEval, a fine-grained framework that evaluates keyphrase systems across four dimensions—reference agreement, faithfulness, diversity, and utility—and was shown to correlate better with human preferences than earlier metrics [5]. A sentence-transformers model specialized for phrases, keyphrase-mpnet-v1, was later released on Hugging Face to support such semantic evaluations, mapping phrases to 768-dimensional dense vectors [9]. The SemR-p authors conducted analyses to test the metric’s semantic sensitivity, ranking awareness, and ability to discriminate between different models and datasets [1][2]. They conclude that SemR-p provides a complementary lens alongside traditional lexical and semantic matching metrics, helping to better reflect user-centred notions of relevance [1][4]. The paper does not report results on downstream tasks or production systems, and the metric has not yet been adopted in benchmark evaluations such as SemEval, the long-running series of computational semantic analysis workshops that has expanded from word-sense disambiguation to broader semantic tasks [7].