Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild

A systematic study of 57 machine-learning evaluation harnesses has classified 16,560 operational issues, finding that the largest share of problems — 41.4 percent — occurs during the Specification stage, when external models, datasets, and scoring judges are integrated [1][2]. The research, posted to arXiv, proposes treating “evaluation engineering” as a distinct software-engineering discipline [1][2]. Evaluation harnesses orchestrate model invocation, data loading, metric computation, and result reporting, yet their operational failures have rarely been studied empirically [1][2]. The authors derived a five-stage harness model and tagged every issue by workflow stage and root cause [1][2]. Across all stages, three root causes accounted for 61.7 percent of classified issues: unimplemented features (24.3 percent), documentation gaps (20.3 percent), and missing input validation (17.2 percent) [1][2]. The pattern shifts by stage. In the Provisioning stage, environment incompatibility and external dependency breakage together represent 36.2 percent of issues, while in the Assessment stage algorithmic error (25.9 percent) and validation gaps (22.5 percent) dominate [1][2]. These findings arrive as the broader ML ecosystem grapples with reliability concerns that extend beyond harnesses. Adversarial-machine-learning research has documented how models can fail when training and test data diverge from the same statistical distribution — a condition often violated in high-stakes applications where users supply fabricated inputs [3]. Convolutional neural networks, still the de-facto standard for computer vision, can be brittle to translation shifts because of downsampling operations, even though their shared-weight architecture was designed to provide translation-equivariant responses [4]. Recurrent neural networks, widely used for sequential data such as speech and time series, have largely been supplanted by transformer architectures that handle long-range dependencies more effectively, though RNNs remain relevant where computational efficiency or real-time processing is critical [5]. The harness study does not address model architecture directly, but its emphasis on specification and assessment failures highlights a parallel risk: even a well-trained model can produce unreliable benchmarks if the evaluation scaffolding is flawed [1][2]. The authors argue that the engineering of that scaffolding — the harness itself — deserves the same rigor applied to model development [1][2].