Abstract
Decision tree learning is one of the most widely used and practical methods for inductive inference. We present a novel method that increases the generalisation of genetically-induced classification trees, which employ linear discriminants as the partitioning function at each internal node. Genetic Programming is employed to search the space of oblique decision trees. At the end of the evolutionary run, a (1+1) Evolution Strategy is used to geometrically optimise the boundaries in the decision space, which are represented by the linear discriminant functions. The evolutionary optimisation concerns maximising the decision-surface margin that is defined to be the smallest distance between the decision-surface and any of the samples. Initial empirical results of the application of our method to a series of datasets from the UCI repository suggest that model generalisation benefits from the margin maximisation, and that the new method is a very competent approach to pattern classification as compared to other learning algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Koza, J.R.: Genetic Programming: on the programming of computers by means of natural selection. MIT Press, Cambridge (1992)
Vladimir, V.: The nature of statistical learning theory, 2nd edn. Springer, Heidelberg (1999)
Koza, J.R.: Concept formation and decision tree induction using the genetic programming paradigm. In: Schwefel, H.-P., Männer, R. (eds.) PPSN 1990. LNCS, vol. 496, pp. 124–128. Springer, Heidelberg (1991)
Folino, G., Pizzuti, C., Spezzano, G.: Genetic Programming and Simulated Annealing: A Hybrid Method to Evolve Decision Trees. In: Poli, R., Banzhaf, W., Langdon, W.B., Miller, J., Nordin, P., Fogarty, T.C. (eds.) EuroGP 2000. LNCS, vol. 1802, pp. 294–303. Springer, Heidelberg (2000)
Eggermont, J.: Evolving Fuzzy Decision Trees with Genetic Programming and Clustering. In: Foster, J.A., Lutton, E., Miller, J., Ryan, C., Tettamanzi, A.G.B. (eds.) EuroGP 2002. LNCS, vol. 2278, pp. 71–82. Springer, Heidelberg (2002)
Rouwhorst, S.E., Engelbrecht, A.P.: Searching the forest: Using decision trees as building blocks for evolutionary search in classification databases. In: Proceedings of the, Congress on Evolutionary Computation CEC 2000, vol. 1, pp. 633–638 (2000)
Bot, M., Langdon, W.B.: Application of genetic programming to induction of linear classification trees. In: Proceedings of the Eleventh Belgium/Netherlands Conference on Artificial Intelligence, BNAIC 1999 (1999)
Marmelstein, R.E., Lamont, G.B.: Pattern classification using a hybrid genetic program decision tree approach. In: Genetic Programming 1998: Proceedings of the Third Annual Conference (1998)
Tsakonas, A.: A comparison of classification accuracy of four genetic programming-evolved intelligent structures. Information Sciences 176(6), 691–724 (2006)
Mugambi, E.M., Hunter, A., Oatley, G., Kennedy, L.: Polynomial-fuzzy decision tree structures for classifying medical data. Knowledge-Based Systems 17(2-4), 81–87 (2004)
Mitchel, T.: Machine Learning. McGraw-Hill, New York (1997)
Estrada-Gil, J.K., Fernandez-Lopez, J.C., Hernandez-Lemus, E., Silva-Zolezzi, I., Hidalgo-Miranda, A., Jimenez-Sanchez, G., Vallejo-Clemente, E.E.: GPDTI: A genetic programming decision tree induction method to find epistatic effects in common complex diseases. Bioinformatics 13(13), i167–i174 (2007)
Kuo, C.-S., Hong, T.-P., Chen, C.-L.: Applying genetic programming technique in classification trees. Soft Computing 11(12), 1165–1172 (2007)
Haruyama, S., Zhao, Q.: Designing smaller decision trees using multiple objective optimization based gps. In: IEEE International Conference on Systems, Man and Cybernetics, vol. 6, p. 5 (2002)
Folino, G., Pizzuti, C., Spezzano, G.: Improving induction decision trees with parallel genetic programming. In: Proceedings 10th Euromicro Workshop on Parallel, Distributed and Network-based Processing, Canary Islands, January 9-11, pp. 181–187. IEEE, Los Alamitos (2002)
Agapitos, A., O’Neill, M., Brabazon, A.: Evolutionary Learning of Technical Trading Rules without Data-Mining Bias. In: Schaefer, R., Cotta, C., Kołodziej, J., Rudolph, G. (eds.) PPSN XI. LNCS, vol. 6238, pp. 294–303. Springer, Heidelberg (2010)
Vapnik, V.N., Chervonenkis, A.Y.: On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications 16(2), 264–280 (1971)
Shawe-Taylor, J., Bartlett, P.L., Williamson, R.C., Anthony, M.: Structural risk minimization over data-dependent hierarchies. IEEE Transactions on Information Theory 44(5) (1998)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)
Newman, D.J., Hettich, S., Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998)
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: an update. SIGKDD Explor. Newsl. 11, 10–18 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Agapitos, A., O’Neill, M., Brabazon, A., Theodoridis, T. (2011). Maximum Margin Decision Surfaces for Increased Generalisation in Evolutionary Decision Tree Learning. In: Silva, S., Foster, J.A., Nicolau, M., Machado, P., Giacobini, M. (eds) Genetic Programming. EuroGP 2011. Lecture Notes in Computer Science, vol 6621. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20407-4_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-20407-4_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-20406-7
Online ISBN: 978-3-642-20407-4
eBook Packages: Computer ScienceComputer Science (R0)