JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

A research team has introduced JADE, a two-layer evaluation framework designed to resolve a persistent tension in assessing AI systems on open-ended professional tasks, according to a paper posted on arXiv [1]. The framework, detailed in a submission last revised on 14 June 2026, aims to balance the rigor of static rubrics with the flexibility of dynamic assessment [1]. The authors, including Lanbo Lin, argue that current methods fall short: fixed rubrics cannot accommodate diverse valid responses, while evaluators that use large language models as judges often exhibit instability and bias [2]. Human experts navigate this by applying domain-grounded principles alongside claim-level analysis, a process JADE seeks to replicate [2]. The framework operates in two layers. The first encodes expert knowledge as a predefined set of evaluation skills to provide stable criteria. The second performs report-specific, claim-level evaluation, incorporating an evidence-dependency gating mechanism that invalidates conclusions built on refuted claims [2]. Experiments on BizBench, a business-task benchmark, showed that JADE improved evaluation stability and surfaced critical agent failure modes that holistic LLM-based evaluators missed [2]. The researchers also reported strong alignment with expert-authored rubrics and demonstrated the framework’s transferability to HealthBench, a benchmark spanning 10 medical and professional domains [2]. The initial preprint was submitted on 6 February 2026 at a size of 4,410 KB, with a revised version posted at 2,210 KB [1]. The paper appears on arXiv, an open-access repository that hosts preprints without peer review and has grown to receive about 24,000 submissions per month as of late 2024 [6]. The field of AI evaluation has drawn broader scrutiny as researchers debate the reliability of machine learning systems. The metascience movement, which studies the practices and incentive structures of research itself, has highlighted widespread methodological flaws and a replication crisis across scientific disciplines [4]. These concerns extend to artificial intelligence, where the challenge of aligning advanced systems with human values has prompted warnings from prominent figures. A 2023 statement signed by hundreds of AI experts and public figures declared that mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war [3]. The JADE framework enters this landscape as a targeted effort to make the evaluation of professional AI agents more reproducible and transparent. The code and data for the project are publicly available on GitHub [2].