A General Framework for Decision Trees via Bregman Divergences

Researchers have proposed two new frameworks: one for decision trees based on Bregman divergences and another for stable low-bit quantization via Hessian conditioning, called HeRo-Q.

A new framework for decision trees, based on Bregman divergences, generalizes the Classification and Regression Trees (CART) paradigm, introduced by Breiman, Friedman, Olshen, and Stone in 1984^[1]. Decision trees are a fundamental tool in statistical learning, and Bregman divergences provide a broad family of loss functions that generalize the squared Euclidean distance. The proposed framework allows for the derivation and interpretation of impurity criteria from common convex and geometric principles. Lev Bregman introduced Bregman divergences in 1967^[1]. In a separate development, researchers propose HeRo-Q, a general framework for stable low-bit quantization via Hessian conditioning. HeRo-Q applies a lightweight, learnable rotation-compression matrix to the weight space prior to quantization, reducing the largest Hessian eigenvalue and enhancing robustness to quantization noise^[2]. HeRo-Q achieves 70.15 GSM8K accuracy on Llama3 8B and requires no architectural modifications, incurring negligible computational overhead. The initial submission size for HeRo-Q is 8,486 KB, and the revised submission size is 2,807 KB^[2].