Hybrid evolutionary learning for synthesizing multi-class pattern recognition systems
Introduction
A pattern recognition system (PRS) is an inherently complex structure consisting of multiple components such as a feature set and a classifier [1]. Generally, the classification accuracy of a PRS is determined by the quality of the feature set. Hand crafting a set of features demands human ingenuity and insight into the problem domain; hence, features are typically standard types of measurements or relatively simple functions of the raw sensor inputs. Humans have difficulty designing elaborate features and often rely on customized computer algorithms or machine-learning techniques such as neural networks and simulated annealing to semi-automate the process of forming features. Most of these techniques still require the designer to specify a parameterized form of the final recognition system, which in-turn reduces the task to a parameter optimization problem. The objective of developing the HELPR system is to provide new tools for automating the design of pattern recognition systems. This paper, after giving an overview of the HELPR architecture, will focus on the HELPR modules that generate complex feature sets from more primitive features.
The philosophy adopted in HELPR research is to spend considerable effort in developing complex features that serve as inputs to a simple classifier backend (see Fig. 1). By using a simple classifier, it is imperative that the features are powerful enough to compensate for the classifiers inability to perform complex mappings. HELPR’s feature representation allows non-linear mappings to be performed. The non-linear features are created using a combination of genetic programming [2], [3] to synthesize arithmetic expressions, genetic algorithms [4] to select a viable set of expressions, and evolutionary programming [5] to optimize parameters within the expressions. The goal is create a compact set of non-linear features that cooperate to solve a multi-class pattern recognition problem.
Fig. 2 provides a system-level view of the HELPR architecture. HELPR is designed for classification of high-dimensional data such as signals or images. These advanced systems use transformations to re-organize the raw data to facilitate the extraction of scalar features by eliminating noisy regions in the data and restructuring regions of the data to highlight areas that are invariant within a given user-defined class. Such transformations are generally represented as mathematical expressions, which are amenable to the techniques of genetic programming. At the beginning of the second stage, the transformed data still has high dimensionality, which must be reduced to be acceptable to traditional classifiers. This reduction must preserve, and possibly enhance, discriminating information present in the transformed data. This process is accomplished by sets of arithmetic expressions, features, that operate on portions of the transformed data. Feature selection is then performed to ensure that the sets are small, yet able to address the classification task. These optimization tasks are amenable to all forms of evolutionary computation. The final set of features is then used by a standard classifier. The entire pipeline of processes is orchestrated by a resource allocation module. This paper, addresses the task of complex feature synthesis and feature selection. Rizki et al. [6] address the evolution of transformations. A later paper will document the integration of all modules.
This paper focuses on the process of complex feature synthesis, and to some extent the process of feature selection. Because of this focus, the system is demonstrated on non-geometric datasets that do not require preliminary transformations. In particular, HELPR is applied to datasets from the University of California’s machine-learning database. Results show that HELPR’s performance meets or exceeds accuracies previously published. When compared against controlled baseline experiments, HELPR’s results were statistically superior in all but one case. In that case, the results were better, but not in a statistically significant way.
Many techniques have been developed for feature extraction from raw input data. These techniques fall into two general categories: feature selection and feature optimization. The techniques that rely on feature selection rapidly generate a large collection of relatively simple features (e.g. random) and then spend considerable time finding a small optimal subset [7], [8], [9], [10]. Subset selection techniques can be successful if the initial pool of detectors contains a set of complementary, yet cooperative, detectors. Basic subset selection techniques do not ordinarily contain control mechanisms to modify the pool of detectors if it is found to be deficient. Rizki and Tamburino [11] adopted a similar approach to synthesize individual features as well as create optimized sets for multi-class pattern recognition.
Alternatively, it is possible to apply standard optimization techniques to the parameters of a fixed set of functional forms defined by an expert. This is the approach adopted in a neural network with a fixed topology [12]. It was also adopted in a system that found the missing templates in a fixed morphological expression [13]. This approach can be efficient and successful if the functional form or network architecture is defined properly; however, it is usually unclear how to systematically define the proper form.
HELPR avoids the problems discussed above by performing feature selection and optimization concurrently. HELPR maintains individual features whose functional forms change to improved system performance. Likewise, sets of features are continually exchanged to identify optimal subsets. An obvious advantage to HELPR is that it has very little human bias in the formation and selection of features. Consequently, non-intuitive regions of the possible solution space are explored. In addition, the HELPR framework is not customized to a particular problem domain and is therefore readily applicable to a wide range of problems.
Section snippets
HELPR’s complex feature synthesis module
Learning systems require a representation to encode the space of potential solutions, a learning strategy that explores the space of possible solutions, and a fitness measure that evaluates points in the search space. Our system uses evolutionary learning (i.e. search strategy) to assemble arithmetic expressions (i.e. representation), which are evaluated using terminal classification accuracy (i.e. performance measure). This section describes these aspects in detail.
Datasets
To test its robustness, HELPR was applied to three recognition problems obtained from the University of California at Irvine (UCI) machine-learning database [19]. This database contains over 100 datasets commonly used in machine-learning research. The main utility of this database is to provide researchers data that can be used to compare the effectiveness of their learning algorithms to others. Datasets were selected based on the following criterion: the problem contained numerical features;
Discussion and conclusions
HELPR is an evolutionary learning system that automatically synthesizes sets of features for multi-class pattern recognition. The system was applied to data taken from the University of California at Irvine machine-learning repository and found to produce better accuracy than what was produced by a baseline system. In addition, the accuracies were better, or comparable, to previously published results on these datasets. These results were obtained using minimal user interaction and without any
Acknowledgements
This work was supported by Air Force Research Laboratory Contract #F33615-99-C-1441.
References (25)
- R. Duda, P. Hart, Pattern Classification and Scene Analysis, Wiley, New York,...
- J.R. Koza, Genetic Programming: On the Programming of Computers By Means of Natural Selection, MIT Press, Cambridge,...
- J.R. Koza, Genetic Programming II: Automatic Discovery of Reusable Programs, MIT Press, Cambridge, MA,...
- J.H. Holland, Adaptation in Natural and Artificial Systems, The University of Michigan Press, Ann Arbor, MI,...
- D.B. Fogel, System Identification Through Simulated Evolution: A Machine Learning Approach to Modeling, Ginn & Co.,...
- M. Rizki, M. Zmuda, L. Tamburino, Evolving pattern recognition systems, IEEE Trans. Evol. Comput., in...
Automatic feature design for optical character recognition using an evolutionary search procedure
IEEE Trans. Pattern Anal. Machine Intell.
(1985)- et al.
Adaptive search for morphological feature detectors
SPIE Image Algebra Morphol. Image Process. I
(1990) Automatic generation of morphological template features
SPIE Image Algebra Morphol. Image Process. I
(1990)- et al.
Automatic feature generation for handwritten digit recognition
IEEE Trans. Pattern Anal. Machine Intell.
(1996)
Cited by (10)
Differential evolution based nearest prototype classifier with optimized distance measures for the features in the data sets
2013, Expert Systems with ApplicationsCitation Excerpt :One of the emerged global optimization methods is the differential evolution (DE) algorithm (Price, Storn, & Lampinen, 2005) that belongs evolutionary computation approaches. DE has also been applied in many areas of pattern recognition, i.e. in remote sensing imagery (Maulik & Saha, 2009), hybrid evolutionary learning in pattern recognition systems (Zmudaa, Rizkib, & Tamburinoc, 2009), in neural network based learning algorithms (Fernandez, Hervas, Martinez, & Cruz, 2009; Magoulas, Plagianakos, & Vrahatis, 2004), in clustering (Bandyopdhyay & Saha, 2007; Das, Abraham, & Konar, 2008; Maulik & Saha, 2010; Omran & Engelbrecht, 2006; Omran, Engelbrecht, & Salman, 2005) and in classification (De Falco, 2012; De Falco, Della Cioppa, & Tarantino, 2006; Triguero, Garcia, & Herrera, 2011) to mention few. Here we tackle classification problems by extending our differential evolution (DE) based classifier (Koloseni, Lampinen, & Luukka, 2012; Luukka & Lampinen, 2010, 2011) further by continuing our generalization on extending it for data featurewise distance measure optimization.
Synergetic use of different evaluation, parameterization and search tools within a multilevel optimization platform
2011, Applied Soft Computing JournalCitation Excerpt :Three multilevel schemes were investigated: On each of their levels, the multilevel evaluation (ME) scheme resorts to different evaluation tools, the multilevel parameterization (MP) handles different problem variants (coarse and fine) and the multilevel search (MS) employs different optimization methods (gradient-based and heuristics). Algorithmic variants that make use of different evaluation software, parameterization or search methods are presented in [5–10]; a detailed survey on hierarchical and multilevel optimization can be found in [2]. The scope of the paper in hand is to present an upgraded multilevel optimization platform in which adjacent levels may differ by more than one tool (evaluation software, parameterization and/or search method).
Affine invariant object shape matching using genetic algorithm with multi-parent orthogonal recombination and migrant principle
2009, Applied Soft Computing JournalNSGA-II/EDA Hybrid Evolutionary Algorithm for Solving Multi-objective Economic/Emission Dispatch Problem
2018, Electric Power Components and SystemsDifferential evolution based multiple vector prototype classifier
2015, Computing and InformaticsGeneralized theory for hybridization of evolutionary algorithms
2014, 2014 IEEE International Conference on Computational Intelligence and Computing Research, IEEE ICCIC 2014