Elsevier

Applied Soft Computing

Volume 2, Issue 4, February 2003, Pages 269-282
Applied Soft Computing

Hybrid evolutionary learning for synthesizing multi-class pattern recognition systems

https://doi.org/10.1016/S1568-4946(02)00060-1Get rights and content

Abstract

This paper describes one aspect of a machine-learning system called HELPR that blends the best aspects of different evolutionary techniques to bootstrap-up a complete recognition system from primitive input data. HELPR uses a multi-faceted representation consisting of a growing sequence of non-linear mathematical expressions. Individual features are represented as tree structures and manipulated using the techniques of genetic programming. Sets of features are represented as list structures that are manipulated using genetic algorithms and evolutionary programming. Complete recognition systems are formed in this version of HELPR by attaching the evolved features to multiple perceptron discriminators. Experiments on datasets from the University of California at Irvine (UCI) machine-learning repository show that HELPR’s performance meets or exceeds accuracies previously published.

Introduction

A pattern recognition system (PRS) is an inherently complex structure consisting of multiple components such as a feature set and a classifier [1]. Generally, the classification accuracy of a PRS is determined by the quality of the feature set. Hand crafting a set of features demands human ingenuity and insight into the problem domain; hence, features are typically standard types of measurements or relatively simple functions of the raw sensor inputs. Humans have difficulty designing elaborate features and often rely on customized computer algorithms or machine-learning techniques such as neural networks and simulated annealing to semi-automate the process of forming features. Most of these techniques still require the designer to specify a parameterized form of the final recognition system, which in-turn reduces the task to a parameter optimization problem. The objective of developing the HELPR system is to provide new tools for automating the design of pattern recognition systems. This paper, after giving an overview of the HELPR architecture, will focus on the HELPR modules that generate complex feature sets from more primitive features.

The philosophy adopted in HELPR research is to spend considerable effort in developing complex features that serve as inputs to a simple classifier backend (see Fig. 1). By using a simple classifier, it is imperative that the features are powerful enough to compensate for the classifiers inability to perform complex mappings. HELPR’s feature representation allows non-linear mappings to be performed. The non-linear features are created using a combination of genetic programming [2], [3] to synthesize arithmetic expressions, genetic algorithms [4] to select a viable set of expressions, and evolutionary programming [5] to optimize parameters within the expressions. The goal is create a compact set of non-linear features that cooperate to solve a multi-class pattern recognition problem.

Fig. 2 provides a system-level view of the HELPR architecture. HELPR is designed for classification of high-dimensional data such as signals or images. These advanced systems use transformations to re-organize the raw data to facilitate the extraction of scalar features by eliminating noisy regions in the data and restructuring regions of the data to highlight areas that are invariant within a given user-defined class. Such transformations are generally represented as mathematical expressions, which are amenable to the techniques of genetic programming. At the beginning of the second stage, the transformed data still has high dimensionality, which must be reduced to be acceptable to traditional classifiers. This reduction must preserve, and possibly enhance, discriminating information present in the transformed data. This process is accomplished by sets of arithmetic expressions, features, that operate on portions of the transformed data. Feature selection is then performed to ensure that the sets are small, yet able to address the classification task. These optimization tasks are amenable to all forms of evolutionary computation. The final set of features is then used by a standard classifier. The entire pipeline of processes is orchestrated by a resource allocation module. This paper, addresses the task of complex feature synthesis and feature selection. Rizki et al. [6] address the evolution of transformations. A later paper will document the integration of all modules.

This paper focuses on the process of complex feature synthesis, and to some extent the process of feature selection. Because of this focus, the system is demonstrated on non-geometric datasets that do not require preliminary transformations. In particular, HELPR is applied to datasets from the University of California’s machine-learning database. Results show that HELPR’s performance meets or exceeds accuracies previously published. When compared against controlled baseline experiments, HELPR’s results were statistically superior in all but one case. In that case, the results were better, but not in a statistically significant way.

Many techniques have been developed for feature extraction from raw input data. These techniques fall into two general categories: feature selection and feature optimization. The techniques that rely on feature selection rapidly generate a large collection of relatively simple features (e.g. random) and then spend considerable time finding a small optimal subset [7], [8], [9], [10]. Subset selection techniques can be successful if the initial pool of detectors contains a set of complementary, yet cooperative, detectors. Basic subset selection techniques do not ordinarily contain control mechanisms to modify the pool of detectors if it is found to be deficient. Rizki and Tamburino [11] adopted a similar approach to synthesize individual features as well as create optimized sets for multi-class pattern recognition.

Alternatively, it is possible to apply standard optimization techniques to the parameters of a fixed set of functional forms defined by an expert. This is the approach adopted in a neural network with a fixed topology [12]. It was also adopted in a system that found the missing templates in a fixed morphological expression [13]. This approach can be efficient and successful if the functional form or network architecture is defined properly; however, it is usually unclear how to systematically define the proper form.

HELPR avoids the problems discussed above by performing feature selection and optimization concurrently. HELPR maintains individual features whose functional forms change to improved system performance. Likewise, sets of features are continually exchanged to identify optimal subsets. An obvious advantage to HELPR is that it has very little human bias in the formation and selection of features. Consequently, non-intuitive regions of the possible solution space are explored. In addition, the HELPR framework is not customized to a particular problem domain and is therefore readily applicable to a wide range of problems.

Section snippets

HELPR’s complex feature synthesis module

Learning systems require a representation to encode the space of potential solutions, a learning strategy that explores the space of possible solutions, and a fitness measure that evaluates points in the search space. Our system uses evolutionary learning (i.e. search strategy) to assemble arithmetic expressions (i.e. representation), which are evaluated using terminal classification accuracy (i.e. performance measure). This section describes these aspects in detail.

Datasets

To test its robustness, HELPR was applied to three recognition problems obtained from the University of California at Irvine (UCI) machine-learning database [19]. This database contains over 100 datasets commonly used in machine-learning research. The main utility of this database is to provide researchers data that can be used to compare the effectiveness of their learning algorithms to others. Datasets were selected based on the following criterion: the problem contained numerical features;

Discussion and conclusions

HELPR is an evolutionary learning system that automatically synthesizes sets of features for multi-class pattern recognition. The system was applied to data taken from the University of California at Irvine machine-learning repository and found to produce better accuracy than what was produced by a baseline system. In addition, the accuracies were better, or comparable, to previously published results on these datasets. These results were obtained using minimal user interaction and without any

Acknowledgements

This work was supported by Air Force Research Laboratory Contract #F33615-99-C-1441.

References (25)

  • R. Duda, P. Hart, Pattern Classification and Scene Analysis, Wiley, New York,...
  • J.R. Koza, Genetic Programming: On the Programming of Computers By Means of Natural Selection, MIT Press, Cambridge,...
  • J.R. Koza, Genetic Programming II: Automatic Discovery of Reusable Programs, MIT Press, Cambridge, MA,...
  • J.H. Holland, Adaptation in Natural and Artificial Systems, The University of Michigan Press, Ann Arbor, MI,...
  • D.B. Fogel, System Identification Through Simulated Evolution: A Machine Learning Approach to Modeling, Ginn & Co.,...
  • M. Rizki, M. Zmuda, L. Tamburino, Evolving pattern recognition systems, IEEE Trans. Evol. Comput., in...
  • F.W.M Stentiford

    Automatic feature design for optical character recognition using an evolutionary search procedure

    IEEE Trans. Pattern Anal. Machine Intell.

    (1985)
  • M Rizki et al.

    Adaptive search for morphological feature detectors

    SPIE Image Algebra Morphol. Image Process. I

    (1990)
  • A.M Gillies

    Automatic generation of morphological template features

    SPIE Image Algebra Morphol. Image Process. I

    (1990)
  • P.D Gader et al.

    Automatic feature generation for handwritten digit recognition

    IEEE Trans. Pattern Anal. Machine Intell.

    (1996)
  • M. Rizki, L. Tamburino, Evolutionary computation applied to pattern recognition, in: Proceedings of the Third Annual...
  • R. Hecht-Nielsen, Neurocomputing, Addison-Wesley, Reading, MA,...
  • Cited by (10)

    • Differential evolution based nearest prototype classifier with optimized distance measures for the features in the data sets

      2013, Expert Systems with Applications
      Citation Excerpt :

      One of the emerged global optimization methods is the differential evolution (DE) algorithm (Price, Storn, & Lampinen, 2005) that belongs evolutionary computation approaches. DE has also been applied in many areas of pattern recognition, i.e. in remote sensing imagery (Maulik & Saha, 2009), hybrid evolutionary learning in pattern recognition systems (Zmudaa, Rizkib, & Tamburinoc, 2009), in neural network based learning algorithms (Fernandez, Hervas, Martinez, & Cruz, 2009; Magoulas, Plagianakos, & Vrahatis, 2004), in clustering (Bandyopdhyay & Saha, 2007; Das, Abraham, & Konar, 2008; Maulik & Saha, 2010; Omran & Engelbrecht, 2006; Omran, Engelbrecht, & Salman, 2005) and in classification (De Falco, 2012; De Falco, Della Cioppa, & Tarantino, 2006; Triguero, Garcia, & Herrera, 2011) to mention few. Here we tackle classification problems by extending our differential evolution (DE) based classifier (Koloseni, Lampinen, & Luukka, 2012; Luukka & Lampinen, 2010, 2011) further by continuing our generalization on extending it for data featurewise distance measure optimization.

    • Synergetic use of different evaluation, parameterization and search tools within a multilevel optimization platform

      2011, Applied Soft Computing Journal
      Citation Excerpt :

      Three multilevel schemes were investigated: On each of their levels, the multilevel evaluation (ME) scheme resorts to different evaluation tools, the multilevel parameterization (MP) handles different problem variants (coarse and fine) and the multilevel search (MS) employs different optimization methods (gradient-based and heuristics). Algorithmic variants that make use of different evaluation software, parameterization or search methods are presented in [5–10]; a detailed survey on hierarchical and multilevel optimization can be found in [2]. The scope of the paper in hand is to present an upgraded multilevel optimization platform in which adjacent levels may differ by more than one tool (evaluation software, parameterization and/or search method).

    • Generalized theory for hybridization of evolutionary algorithms

      2014, 2014 IEEE International Conference on Computational Intelligence and Computing Research, IEEE ICCIC 2014
    View all citing articles on Scopus
    View full text