Experiments with Optimal Model Trees

A new study investigates whether building model trees with globally optimal structures, rather than the typical greedy approach, can yield more accurate and interpretable machine learning models using linear support vector machines at the leaf nodes [1]. Model trees are a form of interpretable machine learning that, unlike classic decision trees with constant leaf values, use linear combinations of predictor variables in their leaves to form predictions [2]. This design can achieve higher accuracy with smaller trees [2]. Standard algorithms for learning these trees operate in a greedy, top-down manner, recursively splitting data into subsets [2]. A key limitation is that the splits chosen are only locally optimal, which can result in trees that are overly complex and less accurate than a tree with a globally optimal structure for the training data [2]. The research, posted on the arXiv pre-print server, presents mixed-integer linear programming formulations to compute these optimal trees [1]. The study, led by Sabino Roselli, was initially submitted in March 2025 and revised through June 2026 [1]. The authors benchmarked their optimal model trees against greedily grown model trees, classic optimal and greedy decision trees, random forests, and support vector machines across a large collection of data sets [2]. The findings indicate that optimal model trees can achieve competitive accuracy with very small trees [2]. The paper also explores replacing axis-parallel splits with multivariate ones, a change that can increase accuracy but at the cost of interpretability [2]. The work is situated within the broader machine learning field, which develops statistical algorithms that learn from data to perform tasks without explicit programming [6]. The computational approach of constructing an optimal tree mirrors challenges in other scientific domains, such as computational phylogenetics, where the main task is to find a phylogenetic tree representing optimal evolutionary ancestry between species [5]. The research was disseminated via arXiv, an open-access repository for electronic preprints in fields including computer science and statistics that, as of late 2024, receives about 24,000 submissions per month [11].