Skip to main content
SpringerLink
Log in
Book cover

European Conference on Genetic Programming

EuroGP 1999: Genetic Programming pp 120–134Cite as

Genetic Programming Discovers Efficient Learning Rules for the Hidden and Output Layers of Feedforward Neural Networks

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 1598))

Abstract

The learning method is critical for obtaining good generalisation in neural networks with limited training data. The Standard BackPropagation (SBP) training algorithm suffers from several problems such as sensitivity to the initial conditions and very slow convergence. The aim of this work is to use Genetic Programming (GP) to discover new supervised learning algorithms which can overcome some of these problems. In previous research a new learning algorithms for the output layer has been discovered using GP. By comparing this with SBP on different problems better performance was demonstrated. This paper shows that GP can also discover better learning algorithms for the hidden layers to be used in conjunction with the algorithm previously discovered. Comparing these with SBP on different problems we show they provide better performances. This study indicates that there exist many supervised learning algorithms better than SBP and that GP can be used to discover them.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. R. Ahmad and F. M. A. Salam. Error back propagation learning using the polynomial energy function. In Proceedings of the International Joint Conference on Neural Networks, Iizuka, Japan, 1992.

    Google Scholar 

  2. Pierre Baldi. Gradient learning algorithm overview: A general dynamical systems perspective. IEEE Transactions on Neural Networks, 6(1):182–195, Jan 1995.

    Article  Google Scholar 

  3. S. Bengio, Y. Bengio, and J. Cloutier. Use of genetic programming for the search of a learning rule for neural networks. In Proceedings of the First Conference on Evolutionary Computation,IEEE World Congress on Computational Intelligence, Orlando-Florida, USA, pages 324–327, 1994.

    Google Scholar 

  4. D. J. Chalmers. The evolution of learning: An experiment in genetic connectionism. In Connectionist Models Summer School. San Mateo, CA., 1990.

    Google Scholar 

  5. X-H Yu G-A Chen. Eficient backpropagation learning using optimal learning rate and momentum. Neural Networks, 10(3):517–527, 1997.

    Article  Google Scholar 

  6. Vladimir Cherkassky and Robert Shepherd. Regularization efiect of weight initialization in back propagation networks. In 1998 IEEE International J. Conference of Neural Networks (IJCNN’98), Anchorage, Alaska, pages 2258–2261. IEEE, May 1998.

    Google Scholar 

  7. C Schittenkopf G Deco and W Brauer. Two strategies to avoid overfitting in feedforward neural networks. Neural Networks, 10(3):505–516, 1997.

    Article  Google Scholar 

  8. D. H. Deterding. Speaker Normalisation for Automatic Speech Recognition. PhD thesis, University of Cambridge, 1989.

    Google Scholar 

  9. A. Harry Eaton and L. Tracy Oliver. Improving the convergence of the back propagation algorithm. Neural Networks, 5:283–288, 1992.

    Article  Google Scholar 

  10. S. E. Fahlman. An empirical study of learning speed in back propagation networks. Technical report, CMU-CS-88-162, Carnegie Mellon University, Pittsburgh, PA., 1988.

    Google Scholar 

  11. M. Gori and A. Tesi. On the problem of local minima in backpropagation. IEEE Transactions on PAMI, 14(1):76–86, 1992.

    Article  Google Scholar 

  12. R. P. Gorman and T.J. Sejnowski. Analysis of hidden units in a layered network trained to classify sonar targets. Neural Networks, 1:75–89, 1988.

    Article  Google Scholar 

  13. S. Haykin. Neural Networks: A Comprehensive Foundation. IEEE Society Press, Macmillan College Publishing, New York 10022, 1994.

    MATH  Google Scholar 

  14. Donald O. Hebb. The Organisation of Behaviour: A Neuropsychological Theory. New York: Wiley, 1949.

    Google Scholar 

  15. Yoshio Hirose, Koichi Yamashit, and Shimpei Hijiya. Back propagation algorithm which varies the number of hidden units. Neural Networks, 4:61–66, 1991.

    Article  Google Scholar 

  16. J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor, Michigan, 1975.

    Google Scholar 

  17. M. J. J. Holt and S. Semnani. Convergence of back-propagation in neural networks using a log-likelihood cost function. Electronics Letters, 26(23):1964–1965, 1990.

    Article  Google Scholar 

  18. Robert A. Jacobs. Increased rates of convergence through learning rate adaptation. Neural Networks, 1:295–307, 1988.

    Article  Google Scholar 

  19. K. Kruschke John and R. Javier Movellan. Fast learning algorithms for neural networks. IEEE Transactions on Circuits and Systems-II: Analogy and Digital Signal Processing, 39(7):453–473, 1992.

    Article  Google Scholar 

  20. H. Kitano. Neurogenetic learning: an integrated method of designing and training neural networks using genetic algorithms. Physica D, 75:225–238, 1994.

    Article  MATH  Google Scholar 

  21. John R. Koza. Genetic Programming: on the programming of computers by means of natural selection. MIT Press, Cambridge, Massachussetts, 1992.

    MATH  Google Scholar 

  22. John R. Koza. Genetic Programming II: Automatic discovery of reusable programs. The MIT Press, Cambridge, Massachussetts, 1994.

    MATH  Google Scholar 

  23. John K. Kruschke and Javier R. Movellan. Benefits of gain: Speeded learning and minimal hidden layers in back propagation networks. IEEE Transactions on Systems, Man and Cybernetics., 21(1), 1991.

    Google Scholar 

  24. T Denoeux R Lengelle. Initializing back propagation networks using prototypes. Neural Networks, 6(3):351–363, 1993.

    Article  Google Scholar 

  25. L.E. Scales. Introduction to non-linear optimization. New York: Springer-Verlag, 1985.

    Book  Google Scholar 

  26. R. Linsker. From basic network principles to neural architecture: Emergence of spatial-opponent cells. In Proceedings of the National Acedemy of Science USA, volume 83, pages 7508–7512, 1986.

    Article  Google Scholar 

  27. F. A. lodewyk Wessels and Etienne Barnard. Avoiding false local minima by proper initialisation of connections. IEEE Transactions on Neural Networks, 3(6):899–905, 1992.

    Article  Google Scholar 

  28. Howard Demuth Martin Hagan and Mark Beale. Neural Network Design. PWS Publishing Company, Boston, MA 02116, 96.

    Google Scholar 

  29. Rangachari Anand Kishan Mehrotra Chilukuri Mohan and Sanjay Ranka. An impproved algorithm for neural network classification of imbalanced training sets. IEEE Transactions on Neural Networks, 4(6):962–969, 1993.

    Article  Google Scholar 

  30. Martin F. Moller. A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks, 6:525–533, 1993.

    Article  Google Scholar 

  31. D. J. Montana and L. Davis. Training feedforaward neural networks using genetic algorithms. In Proceedings of Eleventh International Joint Conference on Artificial Intelligence (IJCAI-89), Detroit, MI, pages 762–767. Morgan Kaufmann, Palo Alto, CA, 1989.

    Google Scholar 

  32. Taek Mu and Hui Cheng. Contrast enhancement for backpropagation. IEEE Transactions on Neural Networks, 7(1):515–524, 1996.

    Google Scholar 

  33. A. Van Ooyen and B Nienhuis. Improving the convergence of the back propagation algorithm. Neural Networks, 5:465–471, 1992.

    Article  Google Scholar 

  34. Alexander G. Parlos, B. Fermandez, Amir Atiya, J. Muthusami, and Wei K. Tsai. An accelerated learning algorithm for multilayer perceptron networks. IEEE Transactions on Neural Networks, 5(3):493–497, 1994.

    Article  Google Scholar 

  35. S. J. Perantonis and D. A. Karras. An eficient constrained training algorithm for feedforward networks. IEEE Transactions on Neural Networks, 6(6):237–149, Nov 1995.

    Article  Google Scholar 

  36. J. Pujol and R. Poli. Eficient evolution of asymmetric recurrent neural networks using a pdgp-inspired two-dimensional representation. In the First European Workshop on Genetic Programming(EUROGP’98), Paris, Springer Lectre Notes in Computer Science, volume 1391, pages 130–141, 1998.

    Google Scholar 

  37. J. Pujol and R. Poli. Evolving neural networks using a dual representation with a combined crossover operator. In the IEEE International Conference on Evolutionary Computation (ICEC’98), Anchorage, Alaska, pages 416–421, 1998.

    Google Scholar 

  38. J. Pujol and R. Poli. Evolving the topology and the weights of neural networks using a dual representation. Special Issue on Evolutionary Learning of the Applied Intelligence Journal, 8(1):73–84, 1998.

    Article  Google Scholar 

  39. Amr Radi and Riccardo Poli. Discovery of backpropagation learning rules using genetic programming. In 1998 IEEE International Conference of Evolutionary Computational (ICEC’98), Anchorage, Alaska, pages 371–375. IEEE, May 1998.

    Google Scholar 

  40. Amr Radi and Riccardo Poli. Genetic programming can discover fast and general learning rules for neural networks. In Third Annual Genetic Programming Conference (GP’98), Madison, Wisconsin, pages 314–323. Morgan Kaufmann, July 1998.

    Google Scholar 

  41. Martin Riedmiller. Advanced supervised learning in multi-layer perceptrons from backpropagation to adaptive learning algorithms. Computer Standards and Interfaces Special Issue on Neural Networks, 16(3):265–275, 1994.

    Article  Google Scholar 

  42. Martin Riedmiller and Heinrich Braun. A direct method for faster backpropagation learning: The RPROP Algorithm. IEEE International Conference on Neural Networks 1993 (ICNN93), San Francisco, pages 586–591, 1993.

    Google Scholar 

  43. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Parallel Distributed Processing. MIT Press, Cambridge, MA, 1986.

    Google Scholar 

  44. Dilip Sarkar. Methods to speed up error back propagation learning algorithm. ACM Computing Surveys, 27(4):519–542, 1995.

    Article  MathSciNet  Google Scholar 

  45. Wolfram Schifimann and Merten Joost. Speeding up backpropagation algorithm by using cross-entropy combined with pattern normalization. International Journal of Uncertainity, Fuzziness and Knowledge-based Systems (IJUFKS),, 6(2):177–126, 1998.

    Google Scholar 

  46. Femando F. Silva and Luis B. Almeida. Speeding Backpropagation, pages 151–158. Advanced Neural Computers, Elsevier Science Publishers B.V. (North-Holland), 1990.

    Google Scholar 

  47. Sara A. Solla, Esther Levin, and Michael Fleisher. Accelerated learning in layered neural networks. Complex System, 2:625–640, 1988.

    MathSciNet  MATH  Google Scholar 

  48. Alessandro Sperduti and Antonina Starita. Speed up learning and network optimization with extended back propagation. Neural Networks, 6:365–383, 1993.

    Article  Google Scholar 

  49. Ramana Vitthal P. Sunthar and Ch. Durgaprasada Rao. The generalized proportional-integral-derivative (pid )gradient descent back propagation algorithm. Neural Networks, 8(4):563–569, 1995.

    Article  Google Scholar 

  50. J. G. Taylor. The Promise of neural networks. Springer-Verlag, London, 1993.

    Book  Google Scholar 

  51. Tom Tollenaere. Super SUB:Fast adaptive back propagation with good scaling properties. Neural Networks, 1:561–573, 1990.

    Article  Google Scholar 

  52. W.T. Zink T.P. Vogl, J.K. Zigler and D.L. Alkon. Accelerating the convergence of the backpropagation method. Biological Cybernetics, 59:256–264, Sept. 1988.

    Google Scholar 

  53. Michael K. Weir. A method for self determination of adaptive learning rate in back propagation. Neural Networks, 4:(371–379), 1991.

    Article  Google Scholar 

  54. D. Whitley and T. Hanson. Optimizing neural networks using faster, more accurate genetic search. In J. D. Schafier, editor, Third International Conference on Genetic Algorithms, Georg Mason University,, pages 391–396. Morgan Kaufmann, 1989.

    Google Scholar 

  55. M. Spears William, K. A. De Jong, T. Baeck, David Fogel, and Hugo de Garis. An overview of evolutionary computation. In Proceedings of the European Conference on Machine Learning, pages 442–459, 1993.

    Google Scholar 

  56. Merten Joost Wolfram Schifimann and R. Werner. Optimisation of the backpropagation algorithm for training multilayer perceptrons. Technical report 16/1992, University of Koblenz, Institute of Physics, 1992.

    Google Scholar 

  57. Byoung-Tak Zhang. Accelerated learning by active example selection. International Journal of Neural Systems, 5(1):67–75, 1994.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, The University of Birmingham, Birmingham, B15 2TT, UK

    Amr Radi & Riccardo Poli

Authors
  1. Amr Radi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Riccardo Poli
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Computer Science, University of Birmingham, Egdbaston, Birmingham, B15 2TT, UK

    Riccardo Poli

  2. Department of Physical Resource Theory, Chalmers University of Technology, S-412 96, Göteborg, Sweden

    Peter Nordin

  3. Centrum voorWiskunde en Informatic, Kruislaan 413, 1098, SJ Amsterdam, The Netherlands

    William B. Langdon

  4. Napier University, 219 Colinton Road, Edinburgh, EH14 1DJ, UK

    Terence C. Fogarty

Rights and permissions

Reprints and permissions

Copyright information

© 1999 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Radi, A., Poli, R. (1999). Genetic Programming Discovers Efficient Learning Rules for the Hidden and Output Layers of Feedforward Neural Networks. In: Poli, R., Nordin, P., Langdon, W.B., Fogarty, T.C. (eds) Genetic Programming. EuroGP 1999. Lecture Notes in Computer Science, vol 1598. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-48885-5_10

Download citation

  • DOI: https://doi.org/10.1007/3-540-48885-5_10

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-65899-3

  • Online ISBN: 978-3-540-48885-9

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation