Skip to main content
SpringerLink
Log in

Permutation-Invariant Representation of Neural Networks with Neuron Embeddings

  • Conference paper
  • First Online:
Genetic Programming (EuroGP 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13223))

Included in the following conference series:

Abstract

Neural networks are traditionally represented in terms of their weights. A key property of this representation is that there are multiple representations of a network which can be obtained by permuting the order of the neurons. These representations are generally not compatible between networks, making recombination a challenge for two arbitrary neural networks - an issue known as the “permutation problem” in neuroevolution. This paper proposes an indirect encoding in which a neural network is represented in terms of interactions between neurons rather than explicit weights, and which works for both fully connected and convolutional networks. In addition to reducing the number of free parameters, this encoding is agnostic to the ordering of neurons, bypassing a key problem for direct weight-based representation. This allows us to transplant individual neurons and layers into another network without accounting for the specific ordering of neurons. We show through experiments on the MNIST and CIFAR-10 datasets that this method is capable of representing networks which achieve comparable performance to direct weight representation, and that combining networks this way preserves a larger degree of performance than through direct weight transfer.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bacciu, D., Mandic, D.P.: Tensor decompositions in deep learning. In: Computational Intelligence, p. 10 (2020)

    Google Scholar 

  2. Chen, X., Cheng, X., Mallat, S.: Unsupervised Deep HAAR Scattering on Graphs. In: Advances in Neural Information Processing System, vol. 27. Curran Associates, Inc. (2014)

    Google Scholar 

  3. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1800–1807. IEEE, Honolulu, HI, July 2017. https://doi.org/10.1109/CVPR.2017.195

  4. Coleman, C., et al.: DAWNBench: an end-to-end deep learning benchmark and competition. In: NIPS ML Systems Workshop, p. 10 (2017)

    Google Scholar 

  5. Das, A., Hossain, M.S., Muhammad Abdullah, S., Ul Islam, R.: Permutation free encoding technique for evolving neural networks. In: Sun, F., Zhang, J., Tan, Y., Cao, J., Yu, W. (eds.) ISNN 2008. LNCS, vol. 5263, pp. 255–265. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-87732-5_29

    Chapter  Google Scholar 

  6. Deng, L., Li, G., Han, S., Shi, L., Xie, Y.: Model compression and hardware acceleration for neural networks: a comprehensive survey. Proc. IEEE 108(4), 485–532 (2020). https://doi.org/10.1109/JPROC.2020.2976475

    Article  Google Scholar 

  7. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota, June 2019. https://doi.org/10.18653/v1/N19-1423, https://www.aclweb.org/anthology/N19-1423

  8. Dürr, P., Mattiussi, C., Floreano, D.: Neuroevolution with analog genetic encoding. In: Runarsson, T.P., Beyer, H.-G., Burke, E., Merelo-Guervós, J.J., Whitley, L.D., Yao, X. (eds.) PPSN 2006. LNCS, vol. 4193, pp. 671–680. Springer, Heidelberg (2006). https://doi.org/10.1007/11844297_68

    Chapter  Google Scholar 

  9. Edwards, H., Storkey, A.: Towards a neural statistician. In: 5th International Conference on Learning Representations (ICLR 2017), pp. 1–13 (2017)

    Google Scholar 

  10. Eliasmith, C., Anderson, C.H.: Neural Engineering: Computation, Representation, and Dynamics in Neurobiological Systems. Computational Neuroscience Series, A Bradford Book, Cambridge, MA, USA, October 2002

    Google Scholar 

  11. Fukushima, K., Miyake, S.: Neocognitron: a self-organizing neural network model for a mechanism of visual pattern recognition. In: Amari, S., Arbib, M.A. (eds.) Competition and Cooperation in Neural Nets, vol. 45, pp. 267–285. Springer, Berlin (1982). https://doi.org/10.1007/978-3-642-46466-9_18

    Chapter  Google Scholar 

  12. Gomez, F.J.: Robust Non-Linear Control through Neuroevolution. Ph.D. thesis, University of Texas at Austin, August 2003

    Google Scholar 

  13. Haase, D., Amthor, M.: Rethinking depthwise separable convolutions: how intra-kernel correlations lead to improved mobilenets. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14588–14597. IEEE, Seattle, WA, USA, June 2020. https://doi.org/10.1109/CVPR42600.2020.01461

  14. Hausknecht, M., Khandelwal, P., Miikkulainen, R., Stone, P.: HyperNEAT-GGP: a hyperNEAT-based Atari general game player. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 217–224 (2012)

    Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, June 2016. https://doi.org/10.1109/CVPR.2016.90

  16. Jaderberg, M., Vedaldi, A., Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. In: Proceedings of the British Machine Vision Conference 2014, pp. 88.1–88.13. British Machine Vision Association, Nottingham (2014). https://doi.org/10.5244/C.28.88

  17. Karaletsos, T., Dayan, P., Ghahramani, Z.: Probabilistic meta-representations of neural networks. arXiv:1810.00555, October 2018

  18. Kiers, H.: Towards a standardized notation and terminology in multiway analysis. J. Chemometrics 14, 105–122 (2000). https://doi.org/10.1002/1099-128X(200005/06)14:33.0.CO;2-I

    Article  Google Scholar 

  19. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015)

    Google Scholar 

  20. Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009). https://doi.org/10.1137/07070111X

    Article  MathSciNet  MATH  Google Scholar 

  21. Koutník, J., Cuccu, G., Schmidhuber, J., Gomez, F.: Evolving large-scale neural networks for vision-based reinforcement learning. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, pp. 1061–1068 (2013)

    Google Scholar 

  22. Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images. Technical Report TR-2009 (2009)

    Google Scholar 

  23. LeCun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)

    Article  Google Scholar 

  24. LeCun, Y., Cortes, C., Burges, C.: MNIST handwritten digit database. ATT Labs. 7, 6 (2010). http://yann.lecun.com/exdb/mnist

  25. Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., Teh, Y.W.: Set transformer: a framework for attention-based permutation-invariant neural networks. In: International Conference on Machine Learning, pp. 3744–3753. PMLR, May 2019

    Google Scholar 

  26. Moriarty, D.E., Mikkulainen, R.: Efficient reinforcement learning through symbiotic evolution. Mach. Learn. 22(1), 11–32 (1996). https://doi.org/10.1023/A:1018004120707

    Article  Google Scholar 

  27. Oseledets, I.: Tensor-train decomposition. SIAM J. Sci. Comput. 33, 2295–2317 (2011). https://doi.org/10.1137/090752286

    Article  MathSciNet  MATH  Google Scholar 

  28. Page, D.: How to Train Your ResNet, September 2018

    Google Scholar 

  29. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 (2019)

  30. Reisinger, J., Miikkulainen, R.: Acquiring evolvability through adaptive representations. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation - GECCO 2007, p. 1045. ACM Press, London, England (2007). https://doi.org/10.1145/1276958.1277164

  31. Schmidhuber, J.: Discovering neural nets with low Kolmogorov complexity and high generalization capability. Neural Netw. 10(5), 857–873 (1997)

    Article  Google Scholar 

  32. Schmidhuber, J., Wierstra, D., Gagliolo, M., Gomez, F.: Training recurrent networks by Evolino. Neural Comput. 19(3), 757–779 (2007). https://doi.org/10.1162/neco.2007.19.3.757

    Article  MATH  Google Scholar 

  33. Smith, L.N.: A disciplined approach to neural network hyper-parameters: part 1-learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820 (2018)

  34. Smith, L.N., Topin, N.: Super-convergence: very fast training of neural networks using large learning rates. In: Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, vol. 11006, p. 1100612. International Society for Optics and Photonics (2019)

    Google Scholar 

  35. Stanley, K.O., Clune, J., Lehman, J., Miikkulainen, R.: Designing neural networks through neuroevolution. Nat. Mach. Intell. 1(1), 24–35 (2019)

    Article  Google Scholar 

  36. Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)

    Article  Google Scholar 

  37. Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  38. Stanley, K.O., Miikkulainen, R.: A taxonomy for artificial embryogeny. Artif. Life 9(2), 93–130 (2003)

    Article  Google Scholar 

  39. Tang, Y., Ha, D.: The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning. arXiv:2109.02869, September 2021

  40. Tang, Y., Nguyen, D., Ha, D.: Neuroevolution of self-interpretable agents. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 414–424 (2020)

    Google Scholar 

  41. Tucker, L.R.: Some mathematical notes on three-mode factor analysis. Psychometrika 31(3), 279–311 (1966). https://doi.org/10.1007/BF02289464

    Article  MathSciNet  Google Scholar 

  42. Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, pp. 6000–6010. Curran Associates Inc., Red Hook, NY, USA, December 2017

    Google Scholar 

  43. Wang, Z., Liu, J.C.: Translating math formula images to latex sequences using deep neural networks with sequence-level training (2019)

    Google Scholar 

  44. Yu, X., Liu, T., Wang, X., Tao, D.: On compressing deep models by low rank and sparse decomposition. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 67–76. IEEE, Honolulu, HI, July 2017. https://doi.org/10.1109/CVPR.2017.15

  45. Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Queen’s University, Kingston, ON, K7L 2N8, Canada

    Ryan Zhou, Christian Muise & Ting Hu

Authors
  1. Ryan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christian Muise
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ting Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ryan Zhou .

Editor information

Editors and Affiliations

  1. University of Trieste, Trieste, Italy

    Eric Medvet

  2. Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

    Gisele Pappa

  3. Victoria University of Wellington, Wellington, New Zealand

    Bing Xue

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhou, R., Muise, C., Hu, T. (2022). Permutation-Invariant Representation of Neural Networks with Neuron Embeddings. In: Medvet, E., Pappa, G., Xue, B. (eds) Genetic Programming. EuroGP 2022. Lecture Notes in Computer Science, vol 13223. Springer, Cham. https://doi.org/10.1007/978-3-031-02056-8_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-02056-8_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-02055-1

  • Online ISBN: 978-3-031-02056-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation