SciDef: Datasets and Tools for Automated Definition Extraction from Scientific Literature with LLMs

A new resource suite called SciDef aims to standardize how scientific definitions are extracted from academic papers, providing a benchmark of 268 human-validated definitions and an open-source pipeline built on large language models, according to a paper posted on arXiv [1]. The suite, developed by researchers including Filip Kučera, addresses a persistent problem in scientific literature: concepts are often defined inconsistently across papers, complicating efforts to compare findings and reuse terminology [2]. SciDef includes three main components: DefExtra, a benchmark of 268 human-validated author-stated definitions drawn from 75 academic papers; DefSim, a set of 60 human-labeled definition-pair similarity judgments; and an open LLM-based pipeline that handles PDF preprocessing, text chunking, definition extraction, prompt optimization, and evaluation [2]. To validate the resources, the team benchmarked 16 language models across different prompting strategies and chunking schemes [2]. The strongest set-level configuration achieved a score of 0.397, while the highest-coverage configuration matched at least one prediction to 86.4% of gold definitions, though it also over-generated candidate definitions [2]. The researchers also demonstrated that a natural language inference-based matching metric aligns strongly with human judgments from the DefSim dataset [2]. The work highlights relevance-aware filtering as the key bottleneck preventing fully automatic definition extraction [2]. The code and datasets have been made publicly available on GitHub [2]. Inconsistent terminology is not unique to any single field. For example, in molecular biology, transcription factors are defined as proteins that control the rate of transcription by binding to specific DNA sequences, yet related proteins such as coactivators and chromatin remodelers, which lack DNA-binding domains, are not classified as transcription factors despite their essential roles in gene regulation [7]. This kind of definitional boundary-drawing can vary across subfields and publications, reinforcing the need for tools like SciDef. Similarly, large-scale policy frameworks face definitional challenges. The United Nations' 17 Sustainable Development Goals, adopted in 2015, encompass broad targets such as climate action, quality education, and gender equality, but critics have noted trade-offs between goals and difficulties tracking qualitative indicators [6]. A 2025 UN report found that only 35% of SDG targets were on track or making moderate progress, with 18% moving in reverse, underscoring how definitional ambiguity can hamper measurement and accountability [6]. By providing a standardized benchmark and extraction pipeline, SciDef offers a foundation for more consistent definition-centric literature analysis across disciplines [2].