Image Thresholding: Understanding Bias of Evaluation Metrics towards Specific Evaluation Functions

A new study finds that common image-quality metrics are not neutral judges of segmentation performance but are inherently biased toward specific thresholding methods, raising questions about the reliability of widely used evaluation practices in computer vision. The research, submitted to arXiv on 26 May 2026, examines multilevel image thresholding, a technique used to partition images into distinct regions for applications spanning medical imaging to remote sensing [1][2]. Practitioners often optimize classical objective functions, such as Otsu's between-class variance and Kapur's entropy, using metaheuristic algorithms and then evaluate the results with metrics like the Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR) [1][2]. These evaluations rest on an implicit assumption that SSIM and PSNR provide unbiased measures of segmentation quality [2]. The study tests that assumption by computing the correlation between thresholding objective functions and quality metrics across all possible thresholds for every image in the BSDS500 dataset [1][2]. The results show that Otsu's criterion consistently exhibits high correlation with both SSIM and PSNR, while Kapur's entropy demonstrates weaker and more variable correlation [1][2]. Otsu outperforms Kapur in correlation with PSNR for all images and with SSIM for over 91% of images [1][2]. The authors conclude that an inherent metric-objective-function bias exists [2]. This finding connects to broader concerns about algorithmic bias, which describes systematic and repeatable tendencies in computational systems to privilege one category over another in ways that can produce unfair outcomes [3]. Bias can enter algorithmic systems through design decisions, data selection, or the choice of evaluation criteria, and it has been documented in domains from criminal justice to healthcare [3]. In machine learning, the selection of evaluation metrics is a foundational step that shapes how model performance is understood and compared [5]. The study's authors argue that the field needs more neutral evaluation frameworks and call for extending the analysis to additional thresholding criteria and application domains [2]. The source code for the paper has been made publicly available [2].