RASC+: Retrieval-Constrained LLM Adjudication for Clinical Value Set Authoring

A two-stage approach combining high-recall retrieval with constrained large language model adjudication can substantially improve automated clinical value set completion, according to research posted to arXiv. The method addresses a core limitation of direct zero-shot generation for standardized medical terminology codes. Clinical value sets are collections of standardized terminology codes used in quality measurement, patient cohort identification, and clinical decision support [1]. The Retrieval-Augmented Set Completion (RASC) benchmark previously demonstrated that direct zero-shot generation by large language models is poorly suited to this task because clinical code systems are large, version-controlled, and not reliably memorized by language models [1][2]. The new work, titled RASC+, proposes a stage-wise alternative. In the first stage, candidate-pool construction is optimized for recall. Using Qwen3-based retrieval with vocabulary-aware expansion and code-display rescue retrieval, researchers increased candidate-pool recall from the original RASC baseline of 0.553 to 0.730 on the full 3,744-value-set test split [1][2]. On the held-out-publisher stratum, pool recall reached 0.655 [1][2]. A higher-recall pool alone did not solve the problem. When the original SAPBert cross-encoder was applied to the expanded pool, full-test macro F1 was 0.287 and held-out-publisher macro F1 was 0.233 [1][2]. The second stage replaces that selector with a blinded GPT-5 adjudicator constrained to choose only from the retrieved candidate pool. This lifted full-test macro F1 to 0.549 and held-out-publisher macro F1 to 0.533 [1][2]. The constrained adjudication design preserves an auditable safety property: every returned code must originate from the candidate pool rather than being freely generated by the language model [1][2]. The approach does not require the model to memorize code systems, which can contain thousands of versioned entries across publishers and terminology standards [1]. The benchmark and method focus on value set authoring, a task that underpins real-world health system functions including quality reporting and clinical research cohort identification [1]. The results suggest that retrieval-constrained LLM adjudication can meaningfully improve completion performance while maintaining the transparency needed for clinical applications [1][2].