Skip to main content
SpringerLink
Log in

Evolving the Topology of Large Scale Deep Neural Networks

  • Conference paper
  • First Online:
Book cover Genetic Programming (EuroGP 2018)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10781))

Included in the following conference series:

Abstract

In the recent years Deep Learning has attracted a lot of attention due to its success in difficult tasks such as image recognition and computer vision. Most of the success in these tasks is merit of Convolutional Neural Networks (CNNs), which allow the automatic construction of features. However, designing such networks is not an easy task, which requires expertise and insight. In this paper we introduce DENSER, a novel representation for the evolution of deep neural networks. In concrete we adapt ideas from Genetic Algorithms (GAs) and Grammatical Evolution (GE) to enable the evolution of sequences of layers and their parameters. We test our approach in the well-known image classification CIFAR-10 dataset. The results show that our method: (i) outperforms previous evolutionary approaches to the generations of CNNs; (ii) is able to create CNNs that have state-of-the-art performance while using less prior knowledge (iii) evolves CNNs with novel topologies, unlikely to be designed by hand. For instance, the best performing CNN obtained during evolution has an unexpected structure using six consecutive dense layers. On the CIFAR-10 the best model reports an average error of 5.87% on test data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Assunção, F., Lourenço, N., Machado, P., Ribeiro, B.: Towards the evolution of multi-layered neural networks: a dynamic structured grammatical evolution approach. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, pp. 393–400. ACM, New York (2017). http://doi.acm.org/10.1145/3071178.3071286

  2. Blum, A., Rivest, R.L.: Training a 3-node neural network is NP-complete. In: Proceedings of the 1st International Conference on Neural Information Processing Systems, pp. 494–501. MIT Press (1988)

    Google Scholar 

  3. Buk, Z., Koutník, J., Šnorek, M.: NEAT in HyperNEAT substituted with genetic programming. In: Kolehmainen, M., Toivanen, P., Beliczynski, B. (eds.) ICANNGA 2009. LNCS, vol. 5495, pp. 243–252. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04921-7_25

    Chapter  Google Scholar 

  4. David, O.E., Greental, I.: Genetic algorithms for evolving deep neural networks. In: Proceedings of the Companion Publication of the 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 1451–1452. ACM (2014)

    Google Scholar 

  5. Deb, K., Anand, A., Joshi, D.: A computationally efficient evolutionary algorithm for real-parameter optimization. Evol. Comput. 10(4), 371–395 (2002)

    Article  Google Scholar 

  6. Deng, L., Hinton, G., Kingsbury, B.: New types of deep neural network learning for speech recognition and related applications: an overview. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8599–8603. IEEE (2013)

    Google Scholar 

  7. Farfade, S.S., Saberian, M.J., Li, L.J.: Multi-view face detection using deep convolutional neural networks. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, ICMR 2015, pp. 643–650. ACM, New York (2015). http://doi.acm.org/10.1145/2671188.2749408

  8. Franco, L., Jerez, J.M.: Constructive Neural Networks, vol. 258. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04512-7

    Book  Google Scholar 

  9. Gomez, F., Schmidhuber, J., Miikkulainen, R.: Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. 9(May), 937–965 (2008)

    MathSciNet  MATH  Google Scholar 

  10. Graham, B.: Fractional max-pooling. arXiv preprint arXiv:1412.6071 (2014)

  11. Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649, May 2013

    Google Scholar 

  12. Junyou, B.: Stock price forecasting using PSO-trained neural networks. In: 2007 IEEE Congress on Evolutionary Computation, CEC 2007, pp. 2879–2885. IEEE (2007)

    Google Scholar 

  13. Kim, H.B., Jung, S.H., Kim, T.G., Park, K.H.: Fast learning method for back-propagation neural network by evolutionary adaptation of learning rates. Neurocomputing 11(1), 101–106 (1996)

    Article  MATH  Google Scholar 

  14. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images (2009)

    Google Scholar 

  15. Leung, F.H.F., Lam, H.K., Ling, S.H., Tam, P.K.S.: Tuning of the structure and parameters of a neural network using an improved genetic algorithm. IEEE Trans. Neural Netw. 14(1), 79–88 (2003)

    Article  Google Scholar 

  16. Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks. arXiv preprint arXiv:1703.00548 (2017)

  17. Mishkin, D., Matas, J.: All you need is a good init. arXiv preprint arXiv:1511.06422 (2015)

  18. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  19. Moriarty, D.E., Miikkulainen, R.: Forming neural networks through efficient and adaptive coevolution. Evol. Comput. 5(4), 373–399 (1997)

    Article  Google Scholar 

  20. O’Neil, M., Ryan, C.: Grammatical evolution. In: O’Neil, M., Ryan, C. (eds.) Grammatical Evolution, pp. 33–47. Springer, Boston (2003). https://doi.org/10.1007/978-1-4615-0447-4_4

    Chapter  Google Scholar 

  21. Palmes, P.P., Hayasaka, T., Usui, S.: Evolution and adaptation of neural networks. In: 2003 Proceedings of the International Joint Conference on Neural Networks, vol. 1, pp. 478–483. IEEE (2003)

    Google Scholar 

  22. Parra, J., Trujillo, L., Melin, P.: Hybrid back-propagation training with evolutionary strategies. Soft. Comput. 18(8), 1603–1614 (2014)

    Article  Google Scholar 

  23. Plis, S.M., Hjelm, D.R., Salakhutdinov, R., Allen, E.A., Bockholt, H.J., Long, J.D., Johnson, H.J., Paulsen, J.S., Turner, J.A., Calhoun, V.D.: Deep learning for neuroimaging: a validation study. Front. Neurosci. 8, 229 (2014)

    Article  Google Scholar 

  24. Radi, A., Poli, R.: Discovering efficient learning rules for feedforward neural networks using genetic programming. In: Abraham, A., Jain, L.C., Kacprzyk, J. (eds.) Recent Advances in Intelligent Paradigms and Applications, pp. 133–159. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-7908-1770-6_7

    Chapter  Google Scholar 

  25. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  26. Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., Adams, R.: Scalable Bayesian optimization using deep neural networks. In: International Conference on Machine Learning, pp. 2171–2180 (2015)

    Google Scholar 

  27. Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)

  28. Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)

    Article  Google Scholar 

  29. Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  30. Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, pp. 497–504. ACM, New York (2017). http://doi.acm.org/10.1145/3071178.3071229

  31. Whitley, D., Starkweather, T., Bogart, C.: Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput. 14(3), 347–361 (1990)

    Article  Google Scholar 

  32. Yao, X., Liu, Y.: Evolutionary artificial neural networks that learn and generalise well. In: 1996 IEEE International Conference on Neural Networks, Washington, DC, USA, Volume on Plenary, Panel and Special Sessions, pp. 159–164 (1996)

    Google Scholar 

  33. Zhang, J., Zong, C.: Deep neural networks in machine translation: an overview. IEEE Intell. Syst. 30(5), 16–25 (2015)

    Article  Google Scholar 

Download references

Acknowledgments

This work is partially funded by: Fundação para a Ciência e Tecnologia (FCT), Portugal, under the grant SFRH/BD/114865/2016, and is based upon work from COST Action CA15140: ImAppNIO, supported by COST (European Cooperation in Science and Technology): www.cost.eu. We would also like to thank NVIDIA for providing us Titan X GPUs.

Author information

Authors and Affiliations

  1. CISUC, Department of Informatics Engineering, University of Coimbra, Coimbra, Portugal

    Filipe Assunção, Nuno Lourenço, Penousal Machado & Bernardete Ribeiro

Authors
  1. Filipe Assunção
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nuno Lourenço
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Penousal Machado
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bernardete Ribeiro
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Filipe Assunção .

Editor information

Editors and Affiliations

  1. Universidade Nova de Lisboa, Lisbon, Portugal

    Mauro Castelli

  2. Brno University of Technology, Brno, Czech Republic

    Lukas Sekanina

  3. Victoria University of Wellington, Wellington, New Zealand

    Mengjie Zhang

  4. University of Parma, Parma, Italy

    Stefano Cagnoni

  5. University of Cádiz, Cádiz, Spain

    Pablo García-Sánchez

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Assunção, F., Lourenço, N., Machado, P., Ribeiro, B. (2018). Evolving the Topology of Large Scale Deep Neural Networks. In: Castelli, M., Sekanina, L., Zhang, M., Cagnoni, S., García-Sánchez, P. (eds) Genetic Programming. EuroGP 2018. Lecture Notes in Computer Science(), vol 10781. Springer, Cham. https://doi.org/10.1007/978-3-319-77553-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-77553-1_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-77552-4

  • Online ISBN: 978-3-319-77553-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation