Evolutionary model trees for handling continuous classes in machine learning

Abstract

Model trees are a particular case of decision trees employed to solve regression problems. They have the advantage of presenting an interpretable output, helping the end-user to get more confidence in the prediction and providing the basis for the end-user to have new insight about the data, confirming or rejecting hypotheses previously formed. Moreover, model trees present an acceptable level of predictive performance in comparison to most techniques used for solving regression problems. Since generating the optimal model tree is an NP-Complete problem, traditional model tree induction algorithms make use of a greedy top-down divide-and-conquer strategy, which may not converge to the global optimal solution. In this paper, we propose a novel algorithm based on the use of the evolutionary algorithms paradigm as an alternate heuristic to generate model trees in order to improve the convergence to globally near-optimal solutions. We call our new approach evolutionary model tree induction (E-Motion). We test its predictive performance using public UCI data sets, and we compare the results to traditional greedy regression/model trees induction algorithms, as well as to other evolutionary approaches. Results show that our method presents a good trade-off between predictive performance and model comprehensibility, which may be crucial in many machine learning applications.

Introduction

Model trees are a popular alternative to classical regression methods, presenting good predictive performance and an intuitive interpretable output. Similar to decision/regression trees, they are structured trees that represent graphically if–then–else rules, which seek to extract implicit knowledge from data sets. While decision trees are used to solve classification problems (i.e., the output is a nominal value), both model and regression trees are used to solve regression problems (i.e., the output is a continuous value). The main difference between the model trees and the regression trees is that whereas regression trees have a single value as the output in their leaves, model trees hold linear models used for calculating the final output.

A model tree is composed by non-terminal nodes, each one representing a test over a data set attribute, and linking edges that partition the data according to the test result. In the bottom of the tree, the terminal nodes hold linear regression models built according to the data that reached each given node. Thus, for predicting the target-attribute value for a given data set instance, we follow down the tree from the root node to the bottom, until a terminal node is reached, and then we apply the corresponding linear model.

Model trees are traditionally induced by divide-and-conquer greedy algorithms which are sequential in nature and locally optimal at each node split [10]. Since inducing the best tree is an NP-Complete problem [32], a greedy heuristic may not derive the best tree overall. In addition, recursive partitioning iteratively degrades the quality of the data set for the purpose of statistical inference, because the larger the number of times the data is partitioned, the smaller becomes the data sample that fits the specific split, leading to results without statistical significance and creating a model that overfits the training data [4].

In order to avoid the drawbacks of the greedy tree-induction algorithms, recent works have focused on powerful ensemble methods¹ [18], [30], which attempt to take advantage of the unstable induction of models by growing a forest of trees from the data and later averaging their predictions. While presenting very good predictive performance, ensemble methods fail to produce a single-tree solution, operating in a black-box fashion.

We highlight the importance of validation and interpretation of discovered knowledge in many data mining applications, because comprehensible models can lead to new insights and hypotheses upon the data [15]. Hence, we believe there should be a trade-off between predictive performance and model comprehensibility, so a predictive system can be useful and helpful in real-world applications.

Evolutionary algorithms (EAs) are a solid heuristic able to deal with a variety of optimization problems. An evolutionary approach for producing trees could enhance the chances of converging to global optima, avoiding solutions that get trapped in local-optima and that are too sensitive to small changes in the data. Evolutionary induction of decision trees is well-explored in the research community. Basgalupp et al. [3] proposed an evolutionary algorithm for the induction of decision trees named LEGAL-Tree, which looks for a good trade-off between accuracy and model comprehensibility. Very few works however propose evolving regression/model trees as an alternative to greedy approaches.

We propose a new algorithm based on the EAs paradigm and also on the core idea presented in LEGAL-Tree [3] to deal with regression problems. By evolving model trees with an evolutionary algorithm, we seek to avoid local-optima convergence by performing a robust global search in the space of candidate solutions. Our approach is, to the best of our knowledge, the first EA that evolves model trees with linear-regression models in their leaves. It is also the first EA that evolves model trees through distinct multi-objective optimization strategies, allowing the user to choose between them.

We test the predictive performance and comprehensibility of this new algorithm using UCI regression data sets [13], and we compare the results to those produced by well-known algorithms, such as M5 [28] and REPTree [35], as well as to other evolutionary approaches. Experimentation shows that our approach presents a good trade-off between predictive performance and comprehensibility.

The remaining of this work is organized as follows. We review tree-structured approaches for mining continuous classes in Section 2. Section 3 details our novel algorithm, namely E-Motion. Sections 4 Comparison to greedy approaches, 5 Comparison to evolutionary approaches present the experiments we have executed, comparing E-Motion to greedy and evolutionary approaches, respectively. In Section 6 we discuss important issues of this work such as comprehensibility of the models produced, overfitting, local-optima avoidance and related issues. We end this work with our conclusions and future work directions in Section 7.