The Normalized Maximum Likelihood for Regular Non-Smooth Models: Measure-Theoretic Foundations and Geometric Sampling

Researchers have developed a rigorous framework for computing the Normalized Maximum Likelihood codelength for non-smooth machine learning models, a class that includes widely used estimators like Lasso and Sparse SVMs, according to a paper submitted on 23 May 2026 [1][2]. The Normalized Maximum Likelihood (NML) codelength, also known as stochastic complexity, is a principled criterion for universal coding [1][2]. Previous formulations using the coarea formula allowed its calculation for smooth models, but that framework collapses when applied to the non-smooth estimators common in modern machine learning [1][2]. The new work provides a measure-theoretic foundation for computing the NML for regular path-differentiable Lipschitz (PDL) estimators [1][2]. By applying classical geometric measure theory and linking the coarea formula with conservative Jacobians, the authors prove that stochastic complexity for non-smooth models is well-posed and theoretically consistent with the outputs of modern Automatic Differentiation [1][2]. To compute this quantity exactly, the researchers introduce the Propose-and-Project Metropolis-Hastings (PDL-PPMH) sampler, a geometric Markov chain Monte Carlo algorithm designed to traverse the non-differentiable level sets of the maximum likelihood estimator [1][2]. The algorithm’s components, including a stochastic tangent space proposal and a provably convergent non-smooth projection solver, are theoretically justified [1][2]. The method’s robustness was demonstrated by sampling from a high-dimensional Lasso posterior with P=2000, while also quantifying the computational scaling that governs the trade-off between exactness and mixing time [1][2]. Monte Carlo methods have a long history in computational statistics. Mean-field particle methods, for instance, rely on sequential interacting samples where each particle interacts with the empirical measures of the process, converging to a deterministic distribution as the system size grows [4]. The new PDL-PPMH sampler extends this lineage by operating directly on the challenging geometry of non-smooth optimization landscapes [1][2]. The paper reports that the exact NML criterion provides a highly data-efficient alternative to cross-validation, achieving statistically indistinguishable predictive optima without requiring data splitting [1][2]. This finding carries practical weight in contexts where data is scarce or expensive to partition. The development also opens a path for theoretical analysis of the NML codelength for regular non-smooth models, a domain that had resisted formal treatment [1][2]. The work was posted on arXiv under the machine learning category and is associated with arXivLabs, a framework for community collaborators developing new features on the platform [1].