On Pitfalls of $\textit{RemOve-And-Retrain}$: Data Processing Inequality Perspective

Researchers have raised concerns about the validity of the RemOve-And-Retrain (ROAR) benchmark, widely used to evaluate feature attribution methods in machine learning.

The ROAR benchmark has been found to be susceptible to improvement through post-processing of attribution maps, which can lead to higher ROAR scores without adding information about the decision function^[1]. This is because the data processing inequality states that processing observations cannot increase information content^[2]. Experiments on datasets such as CIFAR-10, SVHN, and CUB-200 have shown a consistent association between blurriness and ROAR performance, indicating a bias toward spatially blurry masks^[1]. Furthermore, the data processing inequality is supported by the optimal Bayes classifier, which proves that processing cannot increase information content^[2]. The findings suggest that an improved ROAR ranking is not necessarily evidence that an attribution map carries more information about the model. Researchers have also noted that low-level tasks, such as denoising and encoding, can be beneficial for classification despite the capabilities of deep neural networks^[2]. Factors such as class separation, training set size, and class balance can affect the gain from pre-classification processing.