A Benchmark Construction and Evaluation Framework for Specialist Domains: Case Study on Defense-related Documents

A new framework called DoRA generates synthetic question-answering benchmarks for specialist fields where labeled data is scarce, using defense documents as its first case study [1]. The Domain-oriented RAG Assessment framework, described in a paper by Bao Doan, addresses what researchers call a cold-start problem: the absence of evaluation benchmarks and post-training data in retrieval-augmented generation for specialist domains [1]. DoRA produces synthetic QA training and test sets with auditable evidence across five domain-specific intents [1]. To avoid circularity, the framework uses different large language model families for training and test splits — Claude Sonnet for training and GPT-4o for test — drawn from disjoint seed-document corpora [1]. When applied to 40 defense-related documents written in English, DoRA yielded approximately 6,600 curated instances [1]. The resulting benchmark comprised 1,259 samples evaluated against eight LLM baselines [1]. A LoRA-adapted Llama3.1-8B model trained on the synthetic set improved performance across six coverage and faithfulness metrics [1]. Under the default GTE retrieval setting, hallucination was reduced by more than half, with gains holding across alternative retrievers and prompting-based baselines [1]. Defense-domain expertise was incorporated at three evaluation stages: assessing the quality of the synthetic QA, determining the reliability of LLM-as-judge scores, and testing generalization on entirely human-written QA examples [1]. The paper positions DoRA as a practical tool for specialist-domain RAG under domain shift, treating defense as a high-stakes case study [1]. Specialist domains such as defense, law, and medicine have long struggled with benchmark scarcity because manual annotation requires costly subject-matter expertise [2]. The DoRA approach sidesteps this by generating labeled data from a small document corpus, a method that could extend to other fields where proprietary or sensitive materials limit open dataset creation [2]. The framework’s use of separate model families for training and evaluation addresses a known weakness in synthetic-data pipelines, where same-model generation can inflate performance estimates [2]. While the paper focuses on defense, the underlying methodology mirrors broader trends in domain adaptation. Islamic finance, for instance, operates under Sharia-compliant rules that prohibit interest and require specialized contractual forms such as mudarabah and murabahah, creating a parallel need for domain-tuned language tools [5]. By 2014, Sharia-compliant financial institutions held roughly $2 trillion in assets, concentrated in Gulf Cooperation Council countries, Pakistan, Iran, and Malaysia, yet the sector still lacks widely adopted NLP benchmarks [5]. DoRA’s document-driven synthetic generation offers a template for such underserved verticals [1][5].