Elsevier

Neurocomputing

Volume 121, 9 December 2013, Pages 274-289
Neurocomputing

Fast learning neural networks using Cartesian genetic programming

https://doi.org/10.1016/j.neucom.2013.04.005Get rights and content

Abstract

A fast learning neuroevolutionary algorithm for both feedforward and recurrent networks is proposed. The method is inspired by the well known and highly effective Cartesian genetic programming (CGP) technique. The proposed method is called the CGP-based Artificial Neural Network (CGPANN). The basic idea is to replace each computational node in CGP with an artificial neuron, thus producing an artificial neural network. The capabilities of CGPANN are tested in two diverse problem domains. Firstly, it has been tested on a standard benchmark control problem: single and double pole for both Markovian and non-Markovian cases. Results demonstrate that the method can generate effective neural architectures in substantially fewer evaluations in comparison to previously published neuroevolutionary techniques. In addition, the evolved networks show improved generalization and robustness in comparison with other techniques. Secondly, we have explored the capabilities of CGPANNs for the diagnosis of Breast Cancer from the FNA (Finite Needle Aspiration) data samples. The results demonstrate that the proposed algorithm gives 99.5% accurate results, thus making it an excellent choice for pattern recognitions in medical diagnosis, owing to its properties of fast learning and accuracy.

The power of a CGP based ANN is its representation which leads to an efficient evolutionary search of suitable topologies. This opens new avenues for applying the proposed technique to other linear/non-linear and Markovian/non-Markovian control and pattern recognition problems.

Introduction

Artificial neural networks (ANNs) not only have the ability to extract information from complex data but also have the potential to resolve linear/non-linear problems in fields such as chemical processes, control, robotics, pattern recognition, computer vision, oil and gas, etc. [1], [2], [3].

Control Systems are conventionally developed by constructing a mathematical model that represents all dynamics of the system [4]. However, some systems that are complex and non-linear cannot be mathematically modeled, as there are no standard and conventional procedures to represent them. Thus the efficient design and development of a controller becomes difficult. Intelligent control is an unconventional process that provides the flexibility of generating abstract models without any indication of hidden dynamics of the system.

One training method used in ANNs is a genetic algorithm. This is broadly known as neuroevolution. For decades, the performance of the neuroevolutionary algorithms has been tested on the non-linear control problem e.g. developing linear and non-linear controller of a standard benchmark problem ‘the inverted pendulum’ for multiple scenarios of poles, optimum localization of mobile robots, auto-pilot helicopter or aircraft controller, automobile crash warning system, rocket control, routing over a data network, coordinating multi-rover systems, time-series prediction, lung sound detection, speech recognition, chemical processes and manufacturing, and the octopus arm task [5], [6], [7], [8], [9], [10], [11]. Many of the developed controllers have been found efficient and useful when compared to the controllers developed by conventional methods.

Genetic Programming is a form of automatic program induction where evolutionary algorithms are used to build computer programs and complex data structures [12], [13], [14]. In this paper, we are using a graph based form of genetic programming called Cartesian genetic programming (CGP) [15], [16], [17]. CGP has been explored in a range of diverse application domain and has shown to produce competitive performance [16], [18], [19], [20], [21], [22]. A powerful aspect of CGP is its representation of graphs coupled with a high degree of genetic redundancy.

We have adapted CGP for representation and evolution of neural networks. We call our neuroevolutionary technique, CGP artificial neural networks (CGPANNs). Both feedforward and recurrent representations are examined. It is applied on a standard benchmark problem—pole balancing ranging from a simple setup to extremely difficult versions. The results obtained demonstrate that the proposed technique has a fast learning capability as it consistently outperforms all the previous methods explored to date.

The CGPANN technique is also applied to the problem of detecting breast cancer. Features from breast mass are extracted using fine needle aspiration (FNA) and the information is applied as input to CGPANN. FNA data available at the Wisconsin Diagnostic Breast Cancer web site is used for training and testing the network capabilities. The system developed produces fast and accurate results when compared to contemporary work done in the field. The error of the model comes out to be as low as 1% for Type-I (wrongly classifying benign samples as malignant) and 0.5% for Type- II (wrongly classifying malignant samples as benign). The paper is organized as follows. Section 2 is an overview of neuroevolution and the different algorithms developed so far. Section 3 describes the background information on Cartesian genetic programming. Section 4 describes the neuroevolutionary algorithm based on Cartesian genetic programming in detail. Section 5 describes the algorithm applied on the standard benchmark problems i.e. single and double pole balancing tasks along with simulation, analysis and results. Section 6 represents in detail the performance of the algorithm on the diagnosis of breast cancer. Section 8 concludes with discussions and future work.

Section snippets

Neuroevolution

Artificial neural networks (ANNs) are computational systems that map complex relationships between inputs and outputs. They are made up of interconnected nodes (neurons) and weighted connections. The properties of these nodes are inspired by biological neurons, and can exhibit complex global behaviour.

A number of evolutionary algorithms have been applied in the past decade to evolve either weights, topology or both of the parameters of artificial neural networks also known as TWEANN (Topology

Cartesian genetic programming

CGP programs are represented in the form of directed acyclic graphs. These graphs are represented as a two dimensional grid of computational nodes. The genes that make up the genotype in CGP are integers that represent where a node gets its data, what operations the node performs on the data, and where the output data required by the user is to be obtained. When the genotype is decoded, some nodes may be ignored. This happens when node outputs are not used in the calculation of output data. We

Cartesian genetic programming-based artificial neural networks

This section describes a neuroevolutionary algorithm that exploits the representation of CGP in generating artificial neural architecture.

Feed forward and recurrent architectures are evolved based on Cartesian genetic programming hereby known as FCGPANN and RCGPANN. Both the architectures follow a direct encoding strategy in which topology, weight and functions are encoded in one genotype. This is evolved to obtain a good topology, possible weights and combination of functions. So the genotype

Case study-I: Pole balancing

Pole balancing is a standard benchmark problem from the field of control theory and artificial neural networks. The aim is to design controllers for unstable and non-linear systems [55], [56]. Single pole balancing consists of a single pole while double pole balancing requires balancing two poles. The poles are attached to the wheeled cart by a hinge. The length of the track the cart can move on is limited to 2.4<x<+2.4. The objective is to apply force ‘F’ to the cart in such a way that the

Case study-II: Application of CGPANN to breast cancer diagnosis

Breast cancer is one of the leading causes of death in women. Detection of the disease at an earlier stage can save precious lives. Different diagnostic tests and procedures are available for it. One of these is the biopsy of the breast, which is quite painful and causes discomfort to the patient but is more reliable than others. Biopsy however is needed in order to ascertain whether a tumor is benign or malignant. Due to the discomfort associated with the biopsy, patients often hesitate to

Further analysis

In order to evaluate and compare CGPANN with other algorithms we arranged the experimental setup such that it complied with the previously published work. In order to do this, we arranged the data for ten (10) fold cross validation. In this method each data set is divided into 10 blocks of approximately equal size. The data is then shuffled to create ten different data sets. At the end the network is trained with the 9 blocks and tested with the tenth (10th) block. After arranging the data, we

Conclusion and future work

In this paper a fast learning neuroevolutionary algorithm based on Cartesian genetic programming (CGPANN) of both feedforward (FCGPANN) and recurrent (RCGPANN) architecture was proposed. The algorithm was tested on the pole-balancing benchmark and for the diagnosis of breast cancer.

The results presented in this paper demonstrate that CGPANN extends the powerful and flexible representation of CGP to the evolution of neural networks. It was observed that FCGPANN and RCGPANN generated solutions

Maryam Mahsal Khan. did her BSc Computer System Engineering from UET Peshawar, Pakistan and a Master from Universti Teknologi Petronas, Malaysia. She is currently a PhD student at the University of Engineering and Technology Peshawar, Pakistan. She has a keen interest in Non-Linear Control, Genetic Algorithms and Genetic Programming, Artificial Neural Networks, Pattern Recognition, Image Processing and Evolvable hardware. She has a range of publications in these fields in the conferences of

References (82)

  • K. Stanley, R. Sherony, N. Kohl, R. Miikkulainen, Neuroevolution of an automobile crash warning system, in: Proceedings...
  • K. Tumer, Coordinating multi-rover systems: evaluation functions for dynamic and noisy environments, in: GECCO-2005,...
  • J. Koutnik, F. Gomez, J. Schmidhuber, Evolving neural networks in compressed weight space, in: GECCO '10, ACM, 2010,...
  • M.M. Khan, G.M. Khan, J.F. Miller, Evolution of optimal ANNs for non-linear control problems using cartesian genetic...
  • F. Gomez et al.

    Accelerated neural evolution through cooperatively coevolved synapses

    J. Mach. Learn. Res.

    (2008)
  • D. Moriarty, Symbiotic Evolution of Neural Networks in Sequential Decision Tasks, Ph.D. Dissertation, University of...
  • N. García-Pedrajas et al.

    Covneta cooperative coevolutionary model for evolving artificial neural networks

    IEEE Trans. Neural Netw.

    (2003)
  • J. Koza

    Genetic ProgrammingOn the Programming of Computers by Means of Natural Selection

    (1992)
  • W. Banzhaf et al.

    Genetic Programming—An Introduction; On the Automatic Evolution of Computer Programs and its Applications

    (1998)
  • R. Poli et al.

    A Field Guide to Genetic Programming

    (2008)
  • J.F. Miller, P. Thomson, T.C. Fogarty, Designing electronic circuits using evolutionary algorithms, arithmetic...
  • J.F. Miller, An empirical study of the efficiency of learning Boolean functions using a cartesian genetic programming...
  • J.F. Miller, P. Thomson, Cartesian genetic programming, in: Proceedings of the 3rd European Conference on Genetic...
  • S. Harding, J.F. Miller, Evolution of robot controller using cartesian genetic programming, in: Genetic Programming,...
  • J.A. Walker, J.F. Miller, Solving real-valued optimisation problems using cartesian genetic programming, in:...
  • J.A. Walker et al.

    Automatic acquisition, evolution and re-use of modules in cartesian genetic programming

    IEEE Trans. Evol. Comput.

    (2008)
  • J.A. Walker et al.

    Parallel evolution using multi-chromosome cartesian genetic programming

    Genet. Program. Evolvable Mach.

    (2009)
  • S.L. Harding et al.

    Developments in cartesian genetic programmingself-modifying cgp

    Genet. Program. Evolvable Mach.

    (2010)
  • J.C. Bongard, R. Pfeifer, Repeated structure and dissociation of genotypic and phenotypic complexity in artificial...
  • C. Lee, J.-H. Kim, Evolutionary ordered neural network with a linked-list encoding scheme, in: Proceedings of IEEE....
  • X. Yao et al.

    Towards designing artificial neural networks by evolution

    Appl. Math. Comput.

    (1996)
  • D. Dasgupta, D. McGregor, Designing application-specific neural networks using the structured genetic algorithm, in:...
  • D.W. Opitz et al.

    Connectionist theory refinementgenetically searching the space of network topologies

    Journal of Artificial Intelligence Research

    (1997)
  • B. Zhang et al.

    Evolving optimal neural networks using genetic algorithms with occam's razor

    Complex Systems

    (1993)
  • V. Maniezzo

    Genetic evolution of the topology and weight distribution of neural networks

    IEEE Trans. Neural Netw.

    (1994)
  • H. Braun, J. Weisbrod, Evolving feedforward neural networks, in: Proceedings of ICANNGA93, Springer-Verlag,...
  • F. Gruau

    Automatic definition of modular neural networks

    Adaptive Behav.

    (1994)
  • G.S. Hornby et al.

    Creating high-level components with a generative representation for body-brain evolution

    Artif. Life

    (2002)
  • M. Mandischer

    Artificial Neural Nets and Genetic Algorithms

    (1993)
  • F. Gruau, D. Whitley, L. Pyeatt, A comparison between cellular encoding and direct encoding for genetic neural...
  • X. Yao, Evolving artificial neural networks, in: Proceedings of the IEEE, vol. 87, no. 9, 1999, pp....
  • Cited by (0)

    Maryam Mahsal Khan. did her BSc Computer System Engineering from UET Peshawar, Pakistan and a Master from Universti Teknologi Petronas, Malaysia. She is currently a PhD student at the University of Engineering and Technology Peshawar, Pakistan. She has a keen interest in Non-Linear Control, Genetic Algorithms and Genetic Programming, Artificial Neural Networks, Pattern Recognition, Image Processing and Evolvable hardware. She has a range of publications in these fields in the conferences of repute.

    Arbab Masood Ahmad. passed his BSc Electrical Engineering in 1990, from the University of Engineering and Technology, Peshawar, Pakistan. He joined Medical Engineering Department of Siemens Pakistan Engineering Company, the same year. He did his MSc Electrical Engineering from the University of Engineering and Technology Taxila, Pakistan in 1997, as part time student. After serving Siemens Pakistan for fourteen years, initially as a Technical service Engineer and later on as Deputy Manager Service, he left the industry and joined the academia. He served COMSATS Institute of Information Technology, Wah, Pakistan, from 2004-2008. He is currently an Assistant Professor in the Department of Computer Systems Engineering, University of Engineering and Technology Peshawar and also doing his PhD from the same university. The title of his research is “Bio-signal Processing using computational intelligence”. He has one journal and three International Conference publications.

    Gul Muhammad Khan. did his BSc Electrical Engineering from UET Peshawar Pakistan with distinction and a PhD in the field of Intelligent System design in a record time of two years and eight months, from the University of York, UK in 2008. Since then he has been working as an Assistant Professor at the University of Engineering and Technology. Creativity and fast execution flow through his veins unquenched. His sharp eye toward innovation and the love he has for creative ideas comes from years of experience as a lead in various research projects and trainings he has had. Being a professional in the field of planning, he has also been a consultant for various electro medical equipment companies. He has a keen interest in Intelligent Smart Grid System Design, Genetic Algorithms and Genetic Programming, Wireless sensor networks and Evolvable hardware. He has a range of publications in these fields both in journals and conferences of repute. He has established a research centre "Center of Intelligent Systems and Network Research" at UET Peshawar, in march 2011, and is the Director of the same since then. Dr. Gul Muhammad Khan introduced two well known algorithms "CGP Developmental Networks" in the field of computational development and "CGP Neural Networks" in the field of Neuroevolution having numerous applications.

    Julian. F. Miller. has a BSc in Physics (Lond), a PhD in Nonlinear Mathematics (City) and a PGCLTHE (Bham) in Teaching. He is an academic in the Department of Electronics at the University of York. He has chaired or co-chaired fifteen international workshops, conferences and conference tracks in Genetic Programming (GP), Evolvable Hardware. He is a former associate editor of IEEE Transactions on Evolutionary Computation and an associate editor of the Journal of Genetic Programming and Evolvable Machines and Natural Computing. He is on the editorial board of the journals: Evolutionary Computation, International Journal of Unconventional Computing and Journal of Natural Computing Research. He has publications in genetic programming, evolutionary computation, quantum computing, artificial life, evolvable hardware, computational development, and nonlinear mathematics. He is a highly cited author with over 4,000 citations and over 210 publications in related areas. He has given nine tutorials on genetic programming and evolvable hardware at leading conferences in evolutionary computation. He received the prestigious EvoStar award in 2011 for outstanding contribution to the field of evolutionary computation. He is the inventor of a highly cited method of genetic programming known as Cartesian Genetic Programming and edited the first book on the subject in 2011.

    View full text