Skip to main content

Advertisement

SpringerLink
Log in

NeuroEvolution: Evolving Heterogeneous Artificial Neural Networks

  • Special Issue
  • Published:
Evolutionary Intelligence Aims and scope Submit manuscript

Abstract

NeuroEvolution is the application of Evolutionary Algorithms to the training of Artificial Neural Networks. Currently the vast majority of NeuroEvolutionary methods create homogeneous networks of user defined transfer functions. This is despite NeuroEvolution being capable of creating heterogeneous networks where each neuron’s transfer function is not chosen by the user, but selected or optimised during evolution. This paper demonstrates how NeuroEvolution can be used to select or optimise each neuron’s transfer function and empirically shows that doing so significantly aids training. This result is important as the majority of NeuroEvolutionary methods are capable of creating heterogeneous networks using the methods described.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

Similar content being viewed by others

Notes

  1. Homogeneous ANNs are ANNs where each neuron’s TF is identical. Heterogeneous ANNs are ANNs where each neuron’s TF is not identical.

  2. Sometimes refereed to as TWEANNs—Topology and Weight Evolving Artificial Neural Networks.

  3. An open source cross platform CGP/CGPANN library has also been recently released for those interested in using CGP or CGPANN [38].

  4. The logistic sigmoid function is often simply referred to as the sigmoid function in the ANN literature. In fact the term sigmoid function refers to any function which is ‘S’ shaped. The logistic sigmoid function is therefore a specific type of sigmoid function along with other functions including the Gompertz function.

  5. The ‘1’ in ‘Cancer1’ refers to the permutation of the dataset; see [29].

  6. \((1+4)\)-ES has been used previously for CGPANN and is also used here for CNE for simplicity. Additionally as a greedy strategy is used with no crossover maintaining genetic diversity with large populations becomes less significant.

  7. Where probabilistic mutation changes each gene to a new valid value with a given probability.

  8. In the case where training and testing results are given the training result is used for comparisons. This is because no steps to combat over training were made.

References

  1. Angeline P, Saunders G, Pollack J (1994) An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans Neural Netw 5(1):54–65

    Article  Google Scholar 

  2. Augusteijn MF, Harrington TP (2004) Evolving transfer functions for artificial neural networks. Neural Comput Appl 13(1):38–46

    Article  Google Scholar 

  3. Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy layer-wise training of deep networks. Adv Neural Fnfo Process Syst 19:153

    Google Scholar 

  4. Cantú-Paz E, Kamath C (2005) An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Trans Syst Man Cybern Part B Cybern 35(5):915–927

    Article  Google Scholar 

  5. Chalup SK, Wiklendt L (2007) Variations of the two-spiral task. Connect Sci 19(2):183–199

    Article  Google Scholar 

  6. Chebira A, Madani K (2003) Advances in soft computing, vol 19, chap. A Neural network based approach for sensors issued data fusion. Physica, Wien, pp 155–160

  7. Cliff D, Harvey I, Husbands P (1992) Incremental evolution of neural network architectures for adaptive behaviour. In: Proceedings of the European symposium on artificial neural networks (ESANN’93), pp 39–44

  8. Duch W, Jankowski N (1999) Survey of neural transfer functions. Neural Comput Surv 2(1):163–212

    Google Scholar 

  9. Duch W, Jankowski N (2001) Transfer functions: hidden possibilities for better neural networks. In: ESANN, pp 81–94

  10. Duch W, Jankowski N, Maszczyk T (2012) Make it cheap: learning with o (nd) complexity. In: The 2012 international joint conference on neural networks (IJCNN). IEEE, pp 1–4

  11. Floreano D, Dürr P, Mattiussi C (2008) Neuroevolution: from architectures to learning. Evolut Intell 1(1):47–62

    Article  Google Scholar 

  12. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the international conference on artificial intelligence and statistics (AISTATS10). Society for artificial intelligence and statistics

  13. Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(5):359–366

    Article  Google Scholar 

  14. Khan MM, Ahmad MA, Khan MG, Miller JF (2013) Fast learning neural networks using Cartesian Genetic Programming. Neurocomputing 121:274–289

    Article  Google Scholar 

  15. Koutník J, Gomez F, Schmidhuber J (2010) Evolving neural networks in compressed weight space. In: Proceedings of the conference on genetic and evolutionary computation (GECCO-10), pp 619–626

  16. Larochelle H, Bengio Y, Louradour J, Lamblin P (2009) Exploring strategies for training deep neural networks. J Mach Learn Res 10:1–40

    MATH  Google Scholar 

  17. Liu Y, Yao X (1996) Evolutionary design of artificial neural networks with different nodes. In: Proceedings of IEEE international conference on evolutionary computation, 1996, pp 670–675. IEEE

  18. Mangasarian OL, Setiono R, Wolberg WH (1990) Large-scale numerical optimization. In: Coleman TF, Li Y (eds) Pattern recognition via linear programming: theory and application to medical diagnosis. SIAM, Philadelphia, PA, pp 22–31

  19. Manning T, Walsh P (2013) Improving the performance of CGPANN for breast cancer diagnosis using crossover and radial basis functions. In: Evolutionary computation, machine learning and data mining in bioinformatics. Springer, Berlin, pp 165–176

  20. McCulloch W, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biol 5(4):115–133

    MATH  MathSciNet  Google Scholar 

  21. Miller JF (2001) What bloat? Cartesian genetic programming on Boolean problems. In: 2001 Genetic and evolutionary computation conference late breaking papers, pp 295–302

  22. Miller JF (2011) Cartesian genetic programming. Springer, Berlin

    Book  MATH  Google Scholar 

  23. Miller JF, Smith S (2006) Redundancy and computational efficiency in Cartesian genetic programming. IEEE Trans Evolut Comput 10(2):167–174

    Article  Google Scholar 

  24. Miller JF, Thomson P (2000) Cartesian genetic programming. In: Proceedings of the third European conference on genetic programming (EuroGP), vol 1820. Springer, Berlin, pp 121–132

  25. Montana D, VanWyk E, Brinn M, Montana J, Milligan S (2009) Evolution of internal dynamics for neural network nodes. Evolut Intel 1(4):233–251

    Article  Google Scholar 

  26. Park J, Sandberg IW (1991) Universal approximation using radial-basis-function networks. Neural Comput 3(2):246–257

    Article  Google Scholar 

  27. Poli R (1996) Discovery of symbolic, neuro-symbolic and neural networks with parallel distributed genetic programming. Cognitive Science Research Papers University of Birmingham CSRP

  28. Poli R (1999) Parallel distributed genetic programming. New ideas in optimization, advanced topics in computer science, pp 403–431

  29. Prechelt L (1994) Proben1: a set of neural network benchmark problems and benchmarking rules. Fakultät für Informatik, Univ. Karlsruhe, Karlsruhe, Germany. Tech Rep 21:94

    Google Scholar 

  30. Richard K, B., John, M., Nico l N, S (1990) Evolving networks: using the genetic algorithm with connectionist learning. Technical report of cognitive computer science research group, Computer Science and Engineering Department (C-014), University of California at San Diego

  31. Rumelhart DE, Hintont GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323(6088):533–536

    Article  Google Scholar 

  32. Schmidt M, Lipson H (2007) Comparison of tree and graph encodings as function of problem complexity. In: Proceedings of the 9th annual conference on genetic and evolutionary computation. ACM, New York, pp 1674–1679

  33. Silva S, Costa E (2009) Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories. Genetic Progr Evolvable Mach 10(2):141–179

    Article  MathSciNet  Google Scholar 

  34. Smolensky P (1986) Parallel distributed processing: explorations in the microstructure of cognition, chap. Information processing in dynamical systems: foundations of harmony theory. MIT Press, Cambridge, pp 194–281

  35. Stanley K, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evolut Comput 10(2):99–127

    Article  Google Scholar 

  36. Teller A, Veloso M (1996) Symbolic visual learning. In: Ikeuchi K, Veloso M (eds) PADO: a new learning architecture for object recognition. Oxford University Press, pp 81–116

  37. Thrun S, Bala J, Bloedorn E, Bratko I, Cestnik B, Cheng J, De Jong K, Dzeroski S, Fahlman S, Fisher D et al (1991) The monk’s problems a performance comparison of different learning algorithms. Technical report, Carnegie Mellon University

  38. Turner AJ (2014) Cartesian genetic programming library. http://cgplibrary.co.uk/

  39. Turner AJ, Miller JF (2013) Cartesian genetic programming encoded artificial neural networks: a comparison using three benchmarks. In: Proceedings of the conference on genetic and evolutionary computation (GECCO-13), pp 1005–1012

  40. Turner AJ, Miller JF (2013) The importance of topology evolution in NeuroEvolution: a case study using Cartesian genetic programming of artificial neural networks. In: Research and development in intelligent systems XXX. Springer, Berlin, pp 213–226

  41. Turner AJ, Miller JF (2014) Cartesian genetic programming: why no bloat? Genetic programming: 17th European conference, vol 8599., EuroGP-2014, LNCS. Springer, Berlin, pp 193–204

  42. Turner AJ, Miller JF (2014) Recurrent Cartesian genetic programming. In: 13th International conference on parallel problem solving from nature (PPSN 2014), LNCS, vol 8672, pp 476–486

  43. Vargha A, Delaney HD (2000) A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J Educ Behav Stat 25(2):101–132

    Google Scholar 

  44. Vassilev VK, Miller JF (2000) The advantages of landscape neutrality in digital circuit evolution. In: Proceedings on international conference on evolvable systems, LNCS, vol 1801. Springer, Berlin, pp 252–263

  45. Weingaertner D, Tatai VK, Gudwin RR, Von Zuben FJ (2002) Hierarchical evolution of heterogeneous neural networks. In: Proceedings of the 2002 congress on evolutionary computation, 2002. CEC’02, vol 2. IEEE, pp 1775–1780

  46. Wieland A (1991) Evolving neural network controllers for unstable systems. In: IJCNN-91-Seattle international joint conference onNeural networks, 1991, vol 2. IEEE, pp 667–673

  47. Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evolut Comput 1(1):67–82

    Article  Google Scholar 

  48. Yao X (1993) A review of evolutionary artificial neural networks. Int J Intell Syst 8(4):539–567

    Article  Google Scholar 

  49. Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447

    Article  Google Scholar 

  50. Yu T, Miller JF (2001) Neutrality and the evolvability of boolean function landscape. In: Genetic programming, LNCS. Springer, Berlin, pp 204–217

Download references

Acknowledgments

We would like to thank the reviewers for their feedback which helped make a much stronger paper.

Author information

Authors and Affiliations

  1. Intelligent Systems Group, Electronics Department, The University of York, York, UK

    Andrew James Turner & Julian Francis Miller

Authors
  1. Andrew James Turner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Julian Francis Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrew James Turner.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Turner, A.J., Miller, J.F. NeuroEvolution: Evolving Heterogeneous Artificial Neural Networks. Evol. Intel. 7, 135–154 (2014). https://doi.org/10.1007/s12065-014-0115-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12065-014-0115-5

Keywords

Search

Navigation