Fast learning neural networks using Cartesian genetic programming
Introduction
Artificial neural networks (ANNs) not only have the ability to extract information from complex data but also have the potential to resolve linear/non-linear problems in fields such as chemical processes, control, robotics, pattern recognition, computer vision, oil and gas, etc. [1], [2], [3].
Control Systems are conventionally developed by constructing a mathematical model that represents all dynamics of the system [4]. However, some systems that are complex and non-linear cannot be mathematically modeled, as there are no standard and conventional procedures to represent them. Thus the efficient design and development of a controller becomes difficult. Intelligent control is an unconventional process that provides the flexibility of generating abstract models without any indication of hidden dynamics of the system.
One training method used in ANNs is a genetic algorithm. This is broadly known as neuroevolution. For decades, the performance of the neuroevolutionary algorithms has been tested on the non-linear control problem e.g. developing linear and non-linear controller of a standard benchmark problem ‘the inverted pendulum’ for multiple scenarios of poles, optimum localization of mobile robots, auto-pilot helicopter or aircraft controller, automobile crash warning system, rocket control, routing over a data network, coordinating multi-rover systems, time-series prediction, lung sound detection, speech recognition, chemical processes and manufacturing, and the octopus arm task [5], [6], [7], [8], [9], [10], [11]. Many of the developed controllers have been found efficient and useful when compared to the controllers developed by conventional methods.
Genetic Programming is a form of automatic program induction where evolutionary algorithms are used to build computer programs and complex data structures [12], [13], [14]. In this paper, we are using a graph based form of genetic programming called Cartesian genetic programming (CGP) [15], [16], [17]. CGP has been explored in a range of diverse application domain and has shown to produce competitive performance [16], [18], [19], [20], [21], [22]. A powerful aspect of CGP is its representation of graphs coupled with a high degree of genetic redundancy.
We have adapted CGP for representation and evolution of neural networks. We call our neuroevolutionary technique, CGP artificial neural networks (CGPANNs). Both feedforward and recurrent representations are examined. It is applied on a standard benchmark problem—pole balancing ranging from a simple setup to extremely difficult versions. The results obtained demonstrate that the proposed technique has a fast learning capability as it consistently outperforms all the previous methods explored to date.
The CGPANN technique is also applied to the problem of detecting breast cancer. Features from breast mass are extracted using fine needle aspiration (FNA) and the information is applied as input to CGPANN. FNA data available at the Wisconsin Diagnostic Breast Cancer web site is used for training and testing the network capabilities. The system developed produces fast and accurate results when compared to contemporary work done in the field. The error of the model comes out to be as low as 1% for Type-I (wrongly classifying benign samples as malignant) and 0.5% for Type- II (wrongly classifying malignant samples as benign). The paper is organized as follows. Section 2 is an overview of neuroevolution and the different algorithms developed so far. Section 3 describes the background information on Cartesian genetic programming. Section 4 describes the neuroevolutionary algorithm based on Cartesian genetic programming in detail. Section 5 describes the algorithm applied on the standard benchmark problems i.e. single and double pole balancing tasks along with simulation, analysis and results. Section 6 represents in detail the performance of the algorithm on the diagnosis of breast cancer. Section 8 concludes with discussions and future work.
Section snippets
Neuroevolution
Artificial neural networks (ANNs) are computational systems that map complex relationships between inputs and outputs. They are made up of interconnected nodes (neurons) and weighted connections. The properties of these nodes are inspired by biological neurons, and can exhibit complex global behaviour.
A number of evolutionary algorithms have been applied in the past decade to evolve either weights, topology or both of the parameters of artificial neural networks also known as TWEANN (Topology
Cartesian genetic programming
CGP programs are represented in the form of directed acyclic graphs. These graphs are represented as a two dimensional grid of computational nodes. The genes that make up the genotype in CGP are integers that represent where a node gets its data, what operations the node performs on the data, and where the output data required by the user is to be obtained. When the genotype is decoded, some nodes may be ignored. This happens when node outputs are not used in the calculation of output data. We
Cartesian genetic programming-based artificial neural networks
This section describes a neuroevolutionary algorithm that exploits the representation of CGP in generating artificial neural architecture.
Feed forward and recurrent architectures are evolved based on Cartesian genetic programming hereby known as FCGPANN and RCGPANN. Both the architectures follow a direct encoding strategy in which topology, weight and functions are encoded in one genotype. This is evolved to obtain a good topology, possible weights and combination of functions. So the genotype
Case study-I: Pole balancing
Pole balancing is a standard benchmark problem from the field of control theory and artificial neural networks. The aim is to design controllers for unstable and non-linear systems [55], [56]. Single pole balancing consists of a single pole while double pole balancing requires balancing two poles. The poles are attached to the wheeled cart by a hinge. The length of the track the cart can move on is limited to . The objective is to apply force ‘F’ to the cart in such a way that the
Case study-II: Application of CGPANN to breast cancer diagnosis
Breast cancer is one of the leading causes of death in women. Detection of the disease at an earlier stage can save precious lives. Different diagnostic tests and procedures are available for it. One of these is the biopsy of the breast, which is quite painful and causes discomfort to the patient but is more reliable than others. Biopsy however is needed in order to ascertain whether a tumor is benign or malignant. Due to the discomfort associated with the biopsy, patients often hesitate to
Further analysis
In order to evaluate and compare CGPANN with other algorithms we arranged the experimental setup such that it complied with the previously published work. In order to do this, we arranged the data for ten (10) fold cross validation. In this method each data set is divided into 10 blocks of approximately equal size. The data is then shuffled to create ten different data sets. At the end the network is trained with the 9 blocks and tested with the tenth (10th) block. After arranging the data, we
Conclusion and future work
In this paper a fast learning neuroevolutionary algorithm based on Cartesian genetic programming (CGPANN) of both feedforward (FCGPANN) and recurrent (RCGPANN) architecture was proposed. The algorithm was tested on the pole-balancing benchmark and for the diagnosis of breast cancer.
The results presented in this paper demonstrate that CGPANN extends the powerful and flexible representation of CGP to the evolution of neural networks. It was observed that FCGPANN and RCGPANN generated solutions
Maryam Mahsal Khan. did her BSc Computer System Engineering from UET Peshawar, Pakistan and a Master from Universti Teknologi Petronas, Malaysia. She is currently a PhD student at the University of Engineering and Technology Peshawar, Pakistan. She has a keen interest in Non-Linear Control, Genetic Algorithms and Genetic Programming, Artificial Neural Networks, Pattern Recognition, Image Processing and Evolvable hardware. She has a range of publications in these fields in the conferences of
References (82)
- et al.
Mass lesion detection with a fuzzy neural network
Pattern Recognition
(2004) - et al.
Self-organizing map for cluster analysis of a breast cancer database
Artif. Intell. Med.
(2003) - et al.
Breast cancer diagnosis using genetic programming generated feature
Pattern Recognition
(2006) - et al.
Model selection for a medical diagnostic decision support systema breast cancer detection case
Artif. Intell. Med.
(2000) An evolutionary artificial neural networks approach for breast cancer diagnosis
Artif. Intell. Med.
(2002)- et al.
Learning good prototypes for classification using filtering and abstraction of instances
Pattern Recognition
(2002) - et al.
An improved technique in porosity predictiona neural network approach
IEEE Trans. Geosci. Remote Sensing
(1995) - et al.
Nonstationary color tracking for vision-based human-computer interaction
IEEE Trans. Neural Netw.
(2002) - et al.
Neuro-fuzzy hybrid control system of tank level in petroleum plant
IEEE Trans. Fuzzy Syst.
(1996) - et al.
Adaptive Approximation Based ControlUnifying Neural, Fuzzy and Traditional Adaptive Approximation Approaches
(2006)
Accelerated neural evolution through cooperatively coevolved synapses
J. Mach. Learn. Res.
Covneta cooperative coevolutionary model for evolving artificial neural networks
IEEE Trans. Neural Netw.
Genetic ProgrammingOn the Programming of Computers by Means of Natural Selection
Genetic Programming—An Introduction; On the Automatic Evolution of Computer Programs and its Applications
A Field Guide to Genetic Programming
Automatic acquisition, evolution and re-use of modules in cartesian genetic programming
IEEE Trans. Evol. Comput.
Parallel evolution using multi-chromosome cartesian genetic programming
Genet. Program. Evolvable Mach.
Developments in cartesian genetic programmingself-modifying cgp
Genet. Program. Evolvable Mach.
Towards designing artificial neural networks by evolution
Appl. Math. Comput.
Connectionist theory refinementgenetically searching the space of network topologies
Journal of Artificial Intelligence Research
Evolving optimal neural networks using genetic algorithms with occam's razor
Complex Systems
Genetic evolution of the topology and weight distribution of neural networks
IEEE Trans. Neural Netw.
Automatic definition of modular neural networks
Adaptive Behav.
Creating high-level components with a generative representation for body-brain evolution
Artif. Life
Artificial Neural Nets and Genetic Algorithms
Cited by (0)
Maryam Mahsal Khan. did her BSc Computer System Engineering from UET Peshawar, Pakistan and a Master from Universti Teknologi Petronas, Malaysia. She is currently a PhD student at the University of Engineering and Technology Peshawar, Pakistan. She has a keen interest in Non-Linear Control, Genetic Algorithms and Genetic Programming, Artificial Neural Networks, Pattern Recognition, Image Processing and Evolvable hardware. She has a range of publications in these fields in the conferences of repute.
Arbab Masood Ahmad. passed his BSc Electrical Engineering in 1990, from the University of Engineering and Technology, Peshawar, Pakistan. He joined Medical Engineering Department of Siemens Pakistan Engineering Company, the same year. He did his MSc Electrical Engineering from the University of Engineering and Technology Taxila, Pakistan in 1997, as part time student. After serving Siemens Pakistan for fourteen years, initially as a Technical service Engineer and later on as Deputy Manager Service, he left the industry and joined the academia. He served COMSATS Institute of Information Technology, Wah, Pakistan, from 2004-2008. He is currently an Assistant Professor in the Department of Computer Systems Engineering, University of Engineering and Technology Peshawar and also doing his PhD from the same university. The title of his research is “Bio-signal Processing using computational intelligence”. He has one journal and three International Conference publications.
Gul Muhammad Khan. did his BSc Electrical Engineering from UET Peshawar Pakistan with distinction and a PhD in the field of Intelligent System design in a record time of two years and eight months, from the University of York, UK in 2008. Since then he has been working as an Assistant Professor at the University of Engineering and Technology. Creativity and fast execution flow through his veins unquenched. His sharp eye toward innovation and the love he has for creative ideas comes from years of experience as a lead in various research projects and trainings he has had. Being a professional in the field of planning, he has also been a consultant for various electro medical equipment companies. He has a keen interest in Intelligent Smart Grid System Design, Genetic Algorithms and Genetic Programming, Wireless sensor networks and Evolvable hardware. He has a range of publications in these fields both in journals and conferences of repute. He has established a research centre "Center of Intelligent Systems and Network Research" at UET Peshawar, in march 2011, and is the Director of the same since then. Dr. Gul Muhammad Khan introduced two well known algorithms "CGP Developmental Networks" in the field of computational development and "CGP Neural Networks" in the field of Neuroevolution having numerous applications.
Julian. F. Miller. has a BSc in Physics (Lond), a PhD in Nonlinear Mathematics (City) and a PGCLTHE (Bham) in Teaching. He is an academic in the Department of Electronics at the University of York. He has chaired or co-chaired fifteen international workshops, conferences and conference tracks in Genetic Programming (GP), Evolvable Hardware. He is a former associate editor of IEEE Transactions on Evolutionary Computation and an associate editor of the Journal of Genetic Programming and Evolvable Machines and Natural Computing. He is on the editorial board of the journals: Evolutionary Computation, International Journal of Unconventional Computing and Journal of Natural Computing Research. He has publications in genetic programming, evolutionary computation, quantum computing, artificial life, evolvable hardware, computational development, and nonlinear mathematics. He is a highly cited author with over 4,000 citations and over 210 publications in related areas. He has given nine tutorials on genetic programming and evolvable hardware at leading conferences in evolutionary computation. He received the prestigious EvoStar award in 2011 for outstanding contribution to the field of evolutionary computation. He is the inventor of a highly cited method of genetic programming known as Cartesian Genetic Programming and edited the first book on the subject in 2011.