DepthLimited crossover in GP for classifier evolution

Abstract

Genetic Programming (GP) provides a novel way of classification with key features like transparency, flexibility and versatility. Presence of these properties makes GP a powerful tool for classifier evolution. However, GP suffers from code bloat, which is highly undesirable in case of classifier evolution. In this paper, we have proposed an operator named “DepthLimited crossover”. The proposed crossover does not let trees increase in complexity while maintaining diversity and efficient search during evolution. We have compared performance of traditional GP with DepthLimited crossover GP, on data classification problems and found that DepthLimited crossover technique provides compatible results without expanding the search space beyond initial limits. The proposed technique is found efficient in terms of classification accuracy, reduced complexity of population and simplicity of evolved classifiers.

Introduction

Genetic Programming (GP) introduced by Koza (1992) is used to automatically construct and evolve computer programs. This new flexible and interesting technique has solved numerous interesting applications. One of these applications is data classification. The need of data classification arises for the tasks like face recognition, speech recognition, fraud detection and knowledge extraction from databases etc.

Data classification can be defined as assigning class labels to a data instance based upon knowledge gained from previously seen labeled data. Various classification algorithms have been proposed and used for their simplicity, understandability or accuracy. Simpler techniques like decision trees are simple and understandable but applicable to small data sets only. Whereas statistical techniques or neural networks require expertise and the results are not comprehensible.

GP offers a distinctive method of representing classifiers in the form of “Numeric Expression Trees” (Loveard & Ciesielski, 2001), or “Genetic Programming Classifier Expressions” (Kishore, Patnaik, Mani, & Agrawal, 2000). These expression trees are evolved using terminal set containing attribute values and an ephemeral constant, and function set containing arithmetic operators +, −, * and /. For each data instance the tree outputs a real integer. This value is mapped to a class. This value-to-class mapping is simple in case of binary classification where positive value corresponds to one class and negative value to the other. The challenge arises in case of multi-class classification, where a single real output is to be mapped to more than two classes. Several methods have been proposed in this regard. One of these methods assigns different thresholds for different classes, this includes static thresholds (Loveard and Ciesielski, 2001, Yang and Smart, 2004), dynamic thresholds (Yang & Smart, 2004) and slotted thresholds (Yang & Smart, 2004). Another method is binary decomposition proposed by Kishore et al., 2000, Loveard and Ciesielski, 2001. A multi-class problem is decomposed into several binary problems and different classifiers are evolved for each class. The result is as many classifiers as there are classes in the data. In this method only one classifier is supposed to output a positive value, or a conflict arise, for which different conflict resolution methods are present in the literature (Kishore et al., 2000, Muni et al., 2004).

GP is an evolutionary algorithm using the recombination operators like crossover and mutation. Crossover is the method of swapping two subtrees from two parent trees and mutation randomly changes a part of a selected tree. Both these operators play an important role to search through the hyperspace of desired solutions and maintain genetic diversity in GP.

The GP based classification has several advantages over other classification techniques. The GP evolved classifiers are transparent and clearly show the relationship among attributes of data in form of a mathematical equation (Kishore et al., 2000) or rules (Engelbrecht, Rouwhorst, & Scheoman, 2002) depending upon the type of data and primitive set used for classifier evolution. The size and structure of a GP tree is not fixed, which offers a flexible search space to probe for the best classification rule. GP has freedom of expression in every way, until it adds to the classification accuracy and adheres to the initial syntax defined for tree structure. GP is readily applicable to the data in its original form and no transformation of data is required (Kishore et al., 2000). GP has an inherent ability to model the data dependence in its evolving structures without any explicit knowledge of underlying structure of dependencies. GP can eliminate attributes unnecessary for classification task, discarding the need of explicit feature extraction algorithm (Kishore et al., 2000).

In addition to above mentioned benefits, GP has a well known drawback of increase in population complexity during evolution (Poli, Langdon, & McPhee, 2008). That is the average number of nodes in population starts increasing. This increase in tree size is also termed as code bloat or code growth. In case of classifier evolution, one is always interested in evolving simple and comprehensible rules (Carreno, Leguizamon, & Wagner, 2007). The increase in complexity of evolved expressions makes them less understandable. Larger trees tend to over-fit the training data decreasing the testing accuracy (Kotsiantis, 2007) whereas the population is evolved in anticipation of increase in accuracy over unseen data. Moreover, larger and complex population adds more computation time in an already computationally expensive technique. These factors raise the need of evolving simpler classifiers using lesser computation and time.

In this paper we propose a method to overcome this drawback of increase in complexity in terms of average number of nodes by limiting the depth of subtrees for crossover operation (DepthLimited crossover). “Depth of a node is number of edges that need to be traversed to reach the node starting from the tree’s root node (which is assumed to be at depth 0). The depth of a tree is the depth of its deepest leaf” (Poli et al., 2008). The proposed method is efficient in terms of classification accuracy and it does not let the complexity of trees increase during the evolution. In this paper we have

• Presented a new crossover operator

• Eliminated bloat by limiting the search space

• Reduced the complexities of evolved classifiers

• Achieved compatible classification results

After a brief introduction in Section 1, Section 2 discusses work relevant to classification and different crossover techniques used in GP. Section 3 presents and explains the work proposed by authors. Obtained results are organized and discussed in Section 4. Section 5 concludes the findings and highlights possible future work.