Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED

A new paper submitted in 2026 warns that naive use of standard variance estimators with the widely adopted CUPED technique can produce severely misleading results in online A/B tests, even as the method remains the gold standard for data-driven decisions at major technology firms [1]. The paper, posted to arXiv on June 17, systematically addresses five frequently encountered but overlooked questions about Controlled-experiment Using Pre-Experiment Data, or CUPED [1]. CUPED leverages the correlation between pre-experiment data and post-experiment metrics to construct unbiased estimators with lower variance, a technique first proposed by Deng et al. in 2013 [5]. It is now routinely used to guide feature launches, pricing optimization, and user experience enhancements [1]. Among the paper's central findings is that the group-specific adjustment consistently outperforms other alternatives in statistical efficiency and provides narrower uncertainty intervals for relative lift [3][4]. For regression-based frameworks, the authors show that the Huber-White sandwich estimator is indispensable, safeguarding against heteroscedasticity and group imbalance where standard ordinary least squares variance estimators fail to provide valid coverage [3][4]. In multi-arm experiments, the research demonstrates that utilizing the full-sample covariate mean is strictly more efficient than pairwise adjustments [3][4]. The investigation also extends to two-stage sampling designs, where the split-sample estimator maintains its efficiency advantages even under compound stochasticity. However, the authors caution that a variance correction term must be applied to account for the randomness of the initial sampling phase [3][4]. A separate 2025 study on arXiv established a formal bridge between control variates and regression adjustment, recommending that practitioners prioritize control variates with group-specific theta estimates because they balance efficiency and computational feasibility for large-scale online experiments [5]. That work noted that the unadjusted variance estimator for such specifications is conservative or consistent in a design-based framework but systematically underestimates the true variance in a model-based setting [5]. The 2026 paper's recommended methodologies have already been deployed and integrated into ByteDance's experimentation platform [1]. The earlier 2025 study also confirmed that its recommended control variates method had been fully implemented on ByteDance's experimental platform, yielding consistent improvements in testing sensitivity [5].