Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data

Text-classification researchers routinely omit measures of uncertainty when reporting performance metrics such as recall and precision, according to a new evaluation that finds common interval methods can be unreliable under conditions typical of social-science work [1]. The paper, submitted on 24 June 2026 to arXiv, notes that metrics like recall, precision, and F1 are point estimates subject to sampling variation, yet confidence intervals are inconsistently reported alongside them [1]. When intervals are provided, they are often estimated with methods that are not appropriate when labelled datasets are small or performance is high [1]. The authors evaluated interval methods across simulations designed to mirror social-science text classification: small to moderate sample sizes, infrequent constructs, and texts nested within individuals [1]. Default approaches such as the Wald interval and the basic percentile bootstrap were the least accurate, with coverage sometimes falling far below the nominal 95% level [1]. Accuracy improved with the use of Agresti-Coull, Wilson, and Clopper-Pearson intervals, as well as a novel pseudo-count regularized bootstrap that is particularly relevant to the calculation of F1 [1]. Producing high-quality labelled training datasets for supervised algorithms is usually difficult and expensive because of the time needed to label the data, a constraint that makes small-sample inference especially pressing [3]. When texts are nested within individuals, the study demonstrates that adjustment for both effective N and the appropriate degrees of freedom is necessary for producing accurate analytic intervals [1]. Among bootstrap methods, the hierarchical bootstrap was more accurate than the cluster bootstrap when individuals produced a moderate number of texts but became overly conservative when individuals produced only a few [1]. The authors aim to improve the transparency of machine-learning applications and to encourage greater attention to validation sample size at the design stage [1].