Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias

A new statistical procedure aims to estimate how a machine-learning model will perform in a new environment before it is deployed, addressing two common sources of failure simultaneously: covariate shift and selective labeling of outcomes, according to a paper submitted to arXiv in June 2026 [1]. The method, described in a preprint submitted on 12 June 2026, uses a double machine learning framework to estimate the target risk of any black-box prediction model under a general loss function [1][2]. The authors derive a bias-corrected estimator based on the influence function of the target risk and show identification of the estimand under standard assumptions [2]. Model performance frequently degrades when algorithms are moved from training data to real-world settings. Two well-documented causes are covariate shift, where the statistical distribution of input features changes between the source and target environments, and selective labels, where outcome data are observed only for cases that passed through a historical decision process [2]. The new procedure tackles both problems jointly, rather than treating them in isolation [1]. Algorithmic bias has been documented across sectors including healthcare, criminal justice, and hiring, often compounding existing racial, socioeconomic, and gender disparities [3]. A 2021 survey identified multiple forms of bias—historical, representation, and measurement—each capable of producing unfair outcomes when models are deployed without adequate pre-deployment testing [3]. The European Union's Artificial Intelligence Act, proposed in 2021 and adopted in 2024, began to address such risks in legal frameworks [3]. The researchers evaluated their estimator using the eICU electronic health records database, a publicly available critical-care dataset [1][2]. The experiments showed that the proposed method tracked the true target risk more accurately than approaches that addressed only selective labels or only covariate shift, as well as baselines that combined standard plug-in approaches [2]. Machine learning models are typically developed under an empirical risk minimization framework, where performance is measured on held-out data drawn from the same distribution as the training set [4]. That assumption breaks when models encounter new populations or decision contexts. The double machine learning approach offers a way to estimate real-world risk without requiring access to fully labeled target data, which is often unavailable or prohibitively expensive to produce [2][7]. The paper appears on arXiv as a statistics and machine learning preprint. arXiv, operated by Cornell University, hosts open-access preprints across physics, mathematics, computer science, and related fields, and has integrated with platforms such as Hugging Face to allow community-built demos to appear alongside paper abstracts [10][11].