Interpreting "Interpretability" and Explaining "Explainability" in Machine Learning in Physics

A new review argues that interpretability and explainability in machine learning for physics are not automatic features but deliberate modeling choices, reframing how scientists should evaluate AI-driven discoveries [1]. The paper, posted on arXiv, draws a sharp distinction between two often-conflated terms. Interpretability is defined as the structural transparency of a model — the ability to understand or approximate its inner workings. Explainability concerns the scientific content of a model — the ability to map it onto domain knowledge [1]. The authors stress that machine-learned models are subject to the same scientific questions as classical models, differing only in scale [1]. This conceptual work lands amid a broader push for algorithmic transparency. The field of explainable AI (XAI) generally aims to provide humans with intellectual oversight over AI algorithms, countering the "black box" tendency of machine learning where even designers cannot explain a specific decision [7]. XAI research explores methods to make reasoning behind predictions more understandable, addressing user requirements to assess safety and scrutinize automated decisions [7]. The review emphasizes that each concept entails a trade-off: interpretability versus expressivity, and explainability versus adaptability [1]. The authors also highlight the importance of task specification and intervention plans as a core aspect of model design [1]. These considerations arrive as machine learning, a field built on statistical algorithms that learn from data and generalize to unseen tasks, becomes deeply embedded in physics research [9]. Other recent theoretical work illustrates the kind of structural analysis the review advocates. One study derived an analytic formula showing that embedding length in contrastive models naturally encodes semantic properties like concept specificity as a byproduct of training, providing a grounded explanation for a previously heuristic observation [3]. Another demonstrated that the effective dynamics of a genetic algorithm follows clipped gradient descent on the loss, with its slowdown controlled not by the number of parameters but by the effective rank of the Hessian — a finding that may explain why such algorithms scale to large search spaces [5]. The review also distinguishes between intrinsic and post-hoc tools available for achieving interpretability and explainability [1]. This framing aligns with the broader XAI goal of enabling users to confirm existing knowledge, challenge it, and generate new assumptions by explaining what has been done and what information actions are based on [7].