Skip to main content
Log in

Modifying genetic programming for artificial neural network development for data mining

  • Focus
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

The development of artificial neural networks (ANNs) is usually a slow process in which the human expert has to test several architectures until he finds the one that achieves best results to solve a certain problem. However, there are some tools that provide the ability of automatically developing ANNs, many of them using evolutionary computation (EC) tools. One of the main problems of these techniques is that ANNs have a very complex structure, which makes them very difficult to be represented and developed by these tools. This work presents a new technique that modifies genetic programming (GP) so as to correctly and efficiently work with graph structures in order to develop ANNs. This technique also allows the obtaining of simplified networks that solve the problem with a small group of neurons. In order to measure the performance of the system and to compare the results with other ANN development methods by means of evolutionary computation (EC) techniques, several tests were performed with problems based on some of the most used test databases in the Data Mining domain. These comparisons show that the system achieves good results that are not only comparable to those of the already existing techniques but, in most cases, improve them.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alba E, Aldana JF, Troya JM (1993) Fully automatic ANN design: A genetic approach. In: Proc. Int. Workshop Artificial Neural Networks (IWANN’93), Lecture Notes in Computer Science, vol. 686. Berlin, Germany: Springer-Verlag, pp 399–404

  • Alpaydin E (1999) Combined 5 × 2 cv F test for comparing supervised classification learning algorithms. Neural Comput 11: 1885–1892

    Article  Google Scholar 

  • Andersen HC, Tsoi AC (1993) A constructive algorithm for the training of a multilayer perceptron based on the genetic algorithm. Complex syst 7(4): 249–268

    MATH  Google Scholar 

  • Andrews R, Cable R, Diederich J, Geva S, Golea M, Hayward R, Ho-Stuart C, Tickle AB (1996) An evaluation and comparison of techniques for extracting and refining rules from artificial neural networks (QUT NRC Tech. Rep.). Queensland University of Technology, Neurocomputing Research Centre, Queensland

  • Angeline PJ, Suders GM, Pollack JB (1994) An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans Neural Netw 5: 54–65

    Article  Google Scholar 

  • Belew R, McInerney J, Schraudolph N (1991) Evolving networks: using the genetic algorithm with connectioninst learning. In: Proceedings of the second artificial life conference, Addison-Wesley, New York, pp 511–547

  • Bengio S, Bengio Y, Cloutier J, Gecsei J, (1992) On the optimization of a synaptic learning rule. In: Preprints of the conference on optimality in artificial and biological neural networks, University of Texas, Dallas)

  • Bishop CM (1995) Neural networks for pattern recognition. Oxford University Press, New York

    Google Scholar 

  • Bot M (1999) Application of genetic programming to induction of linear classification trees, final term project report. Vrije Universiteit, Amsterdam

    Google Scholar 

  • Cangelosi A, Nolfi S, Parisi D (1994) Cell division and migration in a ‘genotype’ for neural networks. Network Comput Neural Syst 5: 497–515

    Article  MATH  Google Scholar 

  • Cantú-Paz E, Kamath C (2005) An empirical comparison of combinatios of evolutionary algorithms and neural networks for classification problems. IEEE Trans Syst Man Cybern B Cybern 9: 915–927

    Article  Google Scholar 

  • Castillo PA, Arenas MG, Castillo-Valdivieso JJ, Merelo JJ, Prieto A, Romero G (2002) Artificial neural networks desing using evolutionary algorithms. In: Proceedings of the seventh world conference on soft computing

  • Chalmers D (1990) The evolution of learning: an experiment in genetic connectionism. In: Touretzky DS, Elman JL, Hinton GE (eds) Proceedings of the 1990 connectionist models summer school. Morgan Kaufmann, San Mateo, pp 81–90

    Google Scholar 

  • Cramer NL (1985) A representation for the adaptive generation of simple sequential programs. In: Grefenstette (ed) Proceedings of first international conference on genetic algorithms, Carnegie-Mellon University, Pittsburgh, pp 183–187

  • Crosher D (1993) The artificial evolution of a generalized class of adaptive processes. In: Yao X (ed) Preprints of AI’93 workshop on evolutionary computation, pp 18–36

  • Dellaert F, Beer RD (1994) Toward an evolvable model of development for autonomous agent synthesis. In: Brooks R, Maes P (eds) Proceedings of the fourth cenference on artificial life. MIT Press, Cambridge

    Google Scholar 

  • Dietterich TG (1998) Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput 10(7): 1895–1924

    Article  Google Scholar 

  • Dodd N, Macfarlane D, Marland C (1991) Optimization of artificial neural network structure using genetic techniques implemented on multiple transputers. In: Welch P, Stiles D, Kunii TL, Bakkers A (eds)Proceedings of Transputing’91, IOS, Amsterdam, pp 687–700

  • Dorado J (1999) Modelo de un sistema para la selección automática en dominios complejos, con una estrategia cooperativa, de conjuntos de entrenamiento y arquitecturas ideales de redes de neuronas artificiales utilizando algoritmos genéticos. PhD Thesis. University of A Coruña

  • Dorado J, Rabuñal JR, Puertas J, Santos A, Rivero D (2002) Prediction and modelling of the flor of a typical urban basin through genetic programming. Applications of Evolutionary Computing, Proceedings of EvoWorshops 2002: EvoCOP, AvoIASP, EvoSTIM/EvoPLAN

  • Engelbrecht AP, Rouwhorst SE, Schoeman L (2001) A building block approach to genetic programming for rule discovery. Data Mining: a Heuristic Approach. Abbass R, Sarkar C, Newton editors, Idea Group Publishing

  • Fahlman S (1988) Faster-learning variations of back-propagation: an empirical study. In: Touretzky DS, Hinton G, Sejnowski T (eds) Proceedings of the 1988 connectionist models summer school, Morgan Kaufmann, Touretzky, San Mateo, pp 38–51

  • Fan Z, Seo K, Rosenberg RC, Hu J, Goodman ED (2002) exploring multiple design topologies using genetic programming and bond graphs. In: GECCO 2002: Proceedings of the genetic and evolutionary computation conference. Springer, pp 1073–1080

  • Fan Z, Seo K, Rosenberg RC, Hu J, Goodman ED (2003) System-level synthesis of mems via genetic programming and bond graphs. Genet Evol Comput—GECCO-2003 2724: 2058–2071

    Article  Google Scholar 

  • Fisher RA (1936) The use of multiple measurements in taxonomic problems. Annals of Eugenics (pp 179–188)

  • Fogel DB, Wasson EC, Porto VW (1995) A step toward computer-assisted mammography using evolutionary programming and neural networks. Cancer Lett 119(1): 93

    Article  Google Scholar 

  • Frean M (1990) The upstart algorithm: a method for constructing and training feedforward neural networks. Neural Comput 2(2): 198–209

    Article  Google Scholar 

  • Friedberg RM (1958) A learning machine: part I. IBM J Res Dev 2(1): 2–13

    Article  MathSciNet  Google Scholar 

  • Friedberg RM, Dunham B, North JH (1959) A learning machine: Part II. IBM J ResDevel 3(3): 282–287

    MathSciNet  Google Scholar 

  • Fujiki C (1987) Using the genetic algorithm to generate lisp source code to solve the prisoner’s dilemma. International Conference on GAs, pp 236–240

  • Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning. Addison-Wesley, Reading

    MATH  Google Scholar 

  • Greenwood GW (1997) Training partially recurrent neural networks using evolutionary strategies. IEEE Trans Speech Audio Process 5: 192–194

    Article  Google Scholar 

  • Gruau F (1994) Automatic definition of modular neural networks. Adapt Behav 3: 151–183

    Article  Google Scholar 

  • Hancock PJB (1990) GANNET: Design of a neural net for face recognition by genetic algorithm. Center for Cognitive and Computational Neuroscience, Department of Computer Science and Psychology, Stirling University, Stirling, Tech. Rep. CCCN-6, August

  • Harp SA, Samad T, Guha A (1989) Toward the genetic synthesis of neural networks. In: Schafer JD (eds) Procedings of third International Conference on genetic algorithms and their applications. Morgan Kaufmann, San Mateo, pp 360–369

    Google Scholar 

  • Harp SA, Samad T, Guha A (1990) Designing application-specific neural networks using the genetic algorithm. In: Touretzky DS (eds) Advances in neural information processing systems 2. Morgan Kaufamnn, San Mate, pp 447–454

    Google Scholar 

  • Haykin S (1999) Neural Netw. 2. Englewood Cliffs, Prentice Hall

    Google Scholar 

  • Herrera F, Hervás C, Otero J, Sánchez L (2004) Un estudio empírico preliminar sobre los tests estadísticos más habituales en el aprendizaje automático. Giraldez R, Riquelme JC, Aguilar JS (eds.) Tendencias de la Minería de Datos en España, Red Española de Minería de Datos y Aprendizaje (TIC2002-11124-E). pp 403–412

  • Holland JJ (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann Arbor

    Google Scholar 

  • Husbands P, Harvey I, Cliff D, Miller G (1994) The use of genetic algorithms for the development of sensorimotor control systems. In: Gaussier P, Nicoud JD (eds) From perception to action. IEEE Press, Los alamitos

    Google Scholar 

  • Hwang MW, Choi JY, Park J (1997) Evolutionary projection neural networks. In: Proceedings of IEEE International Conference on Evolutionary Computation, ICEC’97, pp 667–671

  • Janson DJ, Frenzel JF (1993) Training product unit neural networks with genetic algorithms. IEEE Exp 8: 26–33

    Article  Google Scholar 

  • Kantschik W, Dittrich P, Brameier M, Banzhaf W (1999) MetaEvolution in graph GP.In: Proceedings of EuroGP’99, LNCS, vol 1598. Springer, Berlin, pp 15–28

  • Kantschik W, Banzhaf W (2002) Linear-graph GP—a new GP structure. In: Proceedings of the 4th European conference on genetic programming, EuroGP 2002

  • Kim H, Jung S, Kim T, Park K (1996) Fast learning method for backpropagation neural network by evolutionary adaptation of learning rates. Neurocomputing 11(1): 101–106

    Article  MATH  Google Scholar 

  • Kim J-H, Choi S-S, Moon B-R (2005) Normalization for neural network in genetic search. Genetic and Evolutionary Computation Conference, pp 1–10

  • Kitano H (1990) Designing neural networks using genetic algorithms with graph generation system. Complex Syst 4: 461–476

    MATH  Google Scholar 

  • Kothari B, Paya B, Esat I (1996) Machinery fault diagnostics using direct encoding graph syntax for optimizing artificial neural network structure. In: Procedings of 1996 3rd biennial Joint Conference on Engineering Systems Design and Analysis, ESDA, Part 7 (of 9), ASME, New York, pp 205–210

  • Koza JR (1992) Genetic programming: on the programming of computers by Jeans of natural selection. MIT Press, Cambridge

    Google Scholar 

  • Liu Y, Yao X (1996) Evolutionary design of artificial neural networks. In: Proceedings of IEEE internationa conference on evolutionary computation (ICEC’96), Nagoya, Japan, pp 670–675

  • Lovell DR, Tsoi AC (1002) The performance of the neocognitron with various S-cell and C-cell transfer functions, Intell. Machines Lab, Dep. Elect. Eng., Univ. Queensland, Tech. Rep., Apr

  • Luke S, Spector L (1996) Evolving graphs and networks with edge encoding: preliminary report. In: Koza J (eds) Late breaking papers at the genetic programming 1996 conference (GP96). Stanford, Stanford Bookstore, pp 117–124

    Google Scholar 

  • Luke S, Spector L (1998) A revised comparison of crossover and mutation in genetic programming. In: Proceedings of the 3rd annual conference on genetic programming. Morgan Kauffman, San Francisco

  • Marin FJ, Sandoval F (1993) Genetic synthesis of discrete-time recurrent neural network. In: Procedings of International workshop artificial neural networks (IWANN’93), Lecture notes in computer science, vol 686. Springer, Berlin, pp 179–184

  • Marshall SJ, Harrison RF (1991) Optimization and training of feedforward neural networks by genetic algorithms. In: Proceedings of the second international conference on artificial neural networks and genetic algorithms, Springer, Berlin, pp 39–43

  • McCulloch WS, Pitts W (1943) A logical calculus of ideas immanent in nervous activity. Bull Math Biophys (5): 115–133

  • Merelo J, Patón M, Canas A, Prieto A, Morán F (1993) Genetic optimization of a multilayer neural network for cluster classification tasks. Neural Netw World 3: 175–186

    Google Scholar 

  • Merrill JWL, Port RF (1991) Fractally configured neural networks. Neural Netw 4(1): 53–60

    Article  Google Scholar 

  • Mertz CJ, Murphy PM (2002) UCI repository of machine learning databases. http://www-old.ics.uci.edu/pub/machine-learning-databases

  • Miller GF, Todd PM, Hedge SU (1989) Designing neural networks using genetic algorithms. In: Proceedings of the third international conference on genetic algorithms. Morgan Kaufmann, San Mateo, pp 379–384

  • Montana DJ (1995) Strongly typed genetic programming. Evol Comput 3(2): 199–200

    Article  Google Scholar 

  • Montana D, David L (1989) Training feed-forward neural networks using genetic algorithms. In: Proceedings of 11th international joint Conference on artificial intelligence. Morgan Kaufmann, San Mateo, pp 762–767

  • Nolfi S, Floreano D (2000) Evolutionary robotics: the biology, intelligence and technology of self-organizing machines. MIT Press/Bradford Books, Cambridge

    Google Scholar 

  • Nolfi S, Parisi D (2002a) Evolution and Learning in neural networks. In: Arbib MA (eds) Handbook of brain theory and neural networks, 2nd edn. MIT Press, Cambridge, pp 415–418

    Google Scholar 

  • Nolfi S, Parisi D (2002b) Evolution of Artificial Neural Networks. In: Arbib MA Handbook of brain theory and neural networks, 2nd edn, MIT Press, Cambridge, 418–421

  • Orchad G (1993) Neural Computing. Research and applications. Ed. Institute of Physics Publishing, Londres

  • Patel D (1996) Using genetic algorithms to construct a network for finantial prediction. In: Proceedings of SPIE: applications of artificial neural networks in image processing, Society of Photo-Optical Instrumentation Engineers, Bellingham, pp 204–213

  • Poli R (1997) Evolution of graph-like programs with parallel distributed genetic programming. Genetic algorithms: proceedings of the Seventh international conference

  • Prechelt L (1996) Early stopping-but when? neural networks: tricks of the trade, pp 55–69

  • Prechelt L (1998) Automatic early stopping using cross validation: qualifying the criteria. Neural Netw 11: 761–767

    Article  Google Scholar 

  • Rabuñal JR, Dorado J, Puertas J, Pazos A, Santos A, Rivero D (2003) Prediction and modelling of the rainfall-runoff transformation of a typical urban basin using ANN and GP. Appl Artifi Intell, vol 17, (4), ISSN 0883-9514 (HID) (I.I 0-615)

  • Rabuñal JR, Dorado J, Pazos A, Pereira J, Rivero D (2004) A new approach to the extraction of ann rules and to their generalization capacity through GP. Neural Comput 16: 1483–1524

    Article  MATH  Google Scholar 

  • Rabuñal JR, Dorado J (eds) (2005) Artificial neural networks in real-life applications. Idea Group Inc

  • Reed R (1993) Pruning algorithms—a survey. IEEE Trans Neural Netw 4(5): 740–747

    Article  Google Scholar 

  • Ripley BD (1996) Pattern recognition and neural networks. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  • Rivero D, Rabuñal JR, Dorado J, Pazos A (2004) Using Genetic Programming for Character Discrimination in Damaged Documents. Applications of Evolutionary Computing, EvoWorkshops2004: EvoBIO, EvoCOMNET, EvoHOT, EvoIASP, EvoMUSART, EvoSTOC (Conference proceedings), p 349–358

  • Rivero D, Rabuñal JR, Dorado J, Pazos A (2005) Time Series Forecast with Anticipation using Genetic Programming. IWANN 2005: 968–975

    Google Scholar 

  • Rivero D, Dorado J, Rabuñal J, Pazos A, Pereira J (2006) Artificial neural network development by means of genetic programming with graph codification, ENFORMATIKA. Trans Eng Comput Technol, World Enformatika Society 15: 209–214

    Google Scholar 

  • Rivero D, Dorado J, Rabuñal J, Pazos A (2006) Using genetic programmning for artificial neural network development and simplification. In: Proceedings of the fifth WSEAS international conference on computational intelligence, Man-Machine Systems and Cybernetics (CIMMACS’06), WSEAS Press, pp 65–71

  • Ribert A, Stocker E, Lecourtier Y, Ennaji A (1994) Optimizing a neural network architecture with an adaptive parameter genetic algorithm. Lecture notes in computer science, vol 1240, Springer, pp 527–535

  • Ritchie MD, White BC, Parker JS, Hahn LW, Moore JH (2003) Optimization of neural network architecture using genetic programming improves detection and modelling of gene-gene interactions in studies of human diseases. BMC Bioinformatics, 3(1)

  • Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. In: Rumelhart DE, McClelland JL (eds) Parallel Distributed processing: explorations in the microstructures of cognition, vol 1. MIT Press, Cambridge, pp 318–362

    Google Scholar 

  • Sietsma J, Dow RJF (1991) Creating Artificial Neural Networks that generalize. Neural Netw 4(1): 67–79

    Article  Google Scholar 

  • Smith RE, Cribbs HB III (1994) Is a learning classifier system a type of neural network. Evol Comput 2(1): 19–36

    Article  Google Scholar 

  • Smith RE, Cribbs IHB (1997) Combined biological paradigms: A neural, genetics-based autonomous systems strategy. Robot Autonomous Syst 22(1): 65–74

    Article  Google Scholar 

  • Stone M (1978) Cross-validation: a review. Matemastische Operationsforschung Statischen. Ser Statist 9: 127–139

    MATH  Google Scholar 

  • Teller A (1996) Evolving programmers: the co-evolution of intelligent recombination operators. In: Angeline P, Kinnear K (eds) Advances in genetic programming II.. MIT Press, Cambridge

    Google Scholar 

  • Teller A, Veloso M (2000) Internal reinforcement in a connectionist genetic programming approach. Artif Intell 120(2): 165–198

    Article  MATH  Google Scholar 

  • Turney P, Whitley D, Anderson R (1996) Special issue on the baldwinian effect. Evol Comput 4(3): 213–329

    Article  Google Scholar 

  • Vonk E, Jain LC, Johnson R (1995) Using genetic algorithms with grammar encoding to generate neural networks. In: Procedings of 1995 IEEE International Conference on Neural Networks, Part 4 (of 6), pp 1928–1931

  • Whitley D, Starkweather T, Bogart C (1990) Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput 14(3): 347–361

    Article  Google Scholar 

  • Yan W, Zhu Z, Hu R (1997) Hybrid genetic/BP algorithm and its application for radar/target classification. In: Proceedings of 1997 IEEE national aerospace and electronics Conference, NAECON. Part 2 (of 2), pp 981–984

  • Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9): 1423–1447

    Article  Google Scholar 

  • Yao X, Shi Y (June 1995) A preliminary study on designing artificial neural networks using co-evolution. In: Proceedings of IEEE Singapore international conference on intelligence control and instrumentation, Singapore, pp 149–154

  • Yao X, Liu Y (1998) Toward designing artificial neural networks by evolution. Appl Math Comput 91(1): 83–90

    Article  MATH  MathSciNet  Google Scholar 

  • Zomorodian A (1995) Context-free language induction by evolution of deterministic push-down automata using genetic programming. In: Working notes of the genetic programming symposium, AAAI-95, AAAI Press, Eric Siegel and John Koza, chairs

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Daniel Rivero.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rivero, D., Dorado, J., Rabuñal, J.R. et al. Modifying genetic programming for artificial neural network development for data mining. Soft Comput 13, 291–305 (2009). https://doi.org/10.1007/s00500-008-0317-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-008-0317-9

Keywords

Navigation