Skip to main content

Advertisement

SpringerLink
Log in

Evolutionary approximation and neural architecture search

  • Published:
Genetic Programming and Evolvable Machines Aims and scope Submit manuscript

Abstract

Automated neural architecture search (NAS) methods are now employed to routinely deliver high-quality neural network architectures for various challenging data sets and reduce the designer’s effort. The NAS methods utilizing multi-objective evolutionary algorithms are especially useful when the objective is not only to minimize the network error but also to reduce the number of parameters (weights) or power consumption of the inference phase. We propose a multi-objective NAS method based on Cartesian genetic programming for evolving convolutional neural networks (CNN). The method allows approximate operations to be used in CNNs to reduce the power consumption of a target hardware implementation. During the NAS process, a suitable CNN architecture is evolved together with selecting approximate multipliers to deliver the best trade-offs between accuracy, network size, and power consumption. The most suitable 8 × N-bit approximate multipliers are automatically selected from a library of approximate multipliers. Evolved CNNs are compared with CNNs developed by other NAS methods on the CIFAR-10 and SVHN benchmark problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://www.fit.vutbr.cz/research/groups/ehw/approxlib/

References

  1. I. Goodfellow, Y. Bengio, A. Courville, Deep Learning (MIT Press, New York, 2016)

    MATH  Google Scholar 

  2. V. Sze, Y. Chen, T. Yang, J.S. Emer, Efficient processing of deep neural networks: a tutorial and survey. Proc. IEEE 105(12), 2295–2329 (2017)

    Article  Google Scholar 

  3. S. Mittal, A survey of techniques for approximate computing. ACM Comput. Surv. 48(4), 1–33 (2016)

    Google Scholar 

  4. P. Panda, A. Sengupta, S.S. Sarwar, G. Srinivasan, S. Venkataramani, A. Raghunathan, K. Roy, Invited—cross-layer approximations for neuromorphic computing: From devices to circuits and systems. In: 53nd Design Automation Conference, pp. 1–6. IEEE (2016). https://doi.org/10.1145/2897937.2905009

  5. S. Venkataramani et al., Efficient AI system design with cross-layer approximate computing. Proc. IEEE 108(12), 2232–2250 (2020). https://doi.org/10.1109/JPROC.2020.3029453

    Article  Google Scholar 

  6. T. Elsken, J.H. Metzen, F. Hutter, Neural architecture search: a survey. J. Mach. Learn. Res. 20(55), 1–21 (2019)

    MathSciNet  MATH  Google Scholar 

  7. P. Ren, Y. Xiao, X. Chang, P.Y. Huang, Z. Li, X. Chen, X. Wang, A comprehensive survey of neural architecture search: challenges and solutions. ACM Comput. Surv. 54, 4 (2021). https://doi.org/10.1145/3447582

    Article  Google Scholar 

  8. B. Zoph, Q.V. Le, Neural architecture search with reinforcement learning (2016). http://arxiv.org/abs/1611.01578

  9. X. Yao, Evolving artificial neural networks. Proc. IEEE 87(9), 1423–1447 (1999)

    Article  Google Scholar 

  10. Z. Lu, I. Whalen, V. Boddeti, Y.D. Dhebar, K. Deb, E.D. Goodman, W. Banzhaf, NSGA-Net: neural architecture search using multi-objective genetic algorithm. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 419–427. ACM (2019)

  11. K.O. Stanley, J. Clune, J. Lehman, R. Miikkulainen, Designing neural networks through neuroevolution. Nat. Mach. Intell. 1, 24–35 (2019)

    Article  Google Scholar 

  12. H. Cai, C. Gan, T. Wang, Z. Zhang, S. Han, Once-for-all: Train one network and specialize it for efficient deployment. In: International Conference on Learning Representations (2020)

  13. H. Cai, L. Zhu, S. Han, ProxylessNAS: Direct neural architecture search on target task and hardware. In: International Conference on Learning Representations (2019)

  14. L. Sekanina, Neural architecture search and hardware accelerator co-search: a survey. IEEE Access 9, 151337–151362 (2021). https://doi.org/10.1109/ACCESS.2021.3126685

    Article  Google Scholar 

  15. V. Mrazek, L. Sekanina, Z. Vasicek, Libraries of approximate circuits: Automated design and application in CNN accelerators. IEEE J. Emerg. Sel. Topics Circuits Syst. 10(4), 406–418 (2020). https://doi.org/10.1109/JETCAS.2020.3032495

  16. S.S. Sarwar, S. Venkataramani, A. Ankit, A. Raghunathan, K. Roy, Energy-efficient neural computing with approximate multipliers. J. Emerg. Technol. Comput. Syst. 14(2), 1–23 (2018)

    Article  Google Scholar 

  17. M. Pinos, V. Mrazek, L. Sekanina, Evolutionary neural architecture search supporting approximate multipliers. In: Genetic Programming—24th European Conference, EuroGP 2021, LNCS, vol. 12691, pp. 82–97. Springer (2021). https://doi.org/10.1007/978-3-030-72812-0_6

  18. V. Mrazek, R. Hrbacek, et al., Evoapprox8b: Library of approximate adders and multipliers for circuit design and benchmarking of approximation methods. In: Proceedings of DATE’17, pp. 258–261 (2017)

  19. M. Abadi, A. Agarwal, et al., TensorFlow: Large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/. Software available from tensorflow.org

  20. F. Vaverka, V. Mrazek, Z. Vasicek, L. Sekanina, TFApprox: Towards a Fast Emulation of DNN Approximate Hardware Accelerators on GPU. In: Design, Automation and Test in Europe, pp. 1–4 (2020)

  21. V. Mrazek, Z. Vasicek, L. Sekanina, A.M. Hanif, M. Shafique, ALWANN: Automatic layer-wise approximation of deep neural network accelerators without retraining. In: Proceedings of the IEEE/ACM International Conference on Computer-Aided Design, pp. 1–8. IEEE (2019)

  22. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385

  23. Intel Movidius vision processing units (VPUs) (2021). https://www.intel.com/content/www/us/en/products/details/processors/movidius-vpu.html

  24. Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang, L. Li, T. Chen, Z. Xu, N. Sun, O. Temam, DaDianNao: a machine-learning supercomputer. In: 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture, pp. 609–622 (2014). https://doi.org/10.1109/MICRO.2014.58

  25. Y. Chen, T. Yang, J. Emer, V. Sze, Eyeriss v2: a flexible accelerator for emerging deep neural networks on mobile devices. IEEE J. Emerg. Select. Topics Circuits Syst. 9(2), 292–308 (2019)

  26. N.P. Jouppi, C. Young, N. Patil, D. Patterson, A domain-specific architecture for deep neural networks. Commun. ACM 61(9), 50–59 (2018)

    Article  Google Scholar 

  27. S. Mittal, A survey of FPGA-based accelerators for convolutional neural networks. Neural Comput. Appl. 32(32), 1109–1139 (2020)

    Article  Google Scholar 

  28. A. Garofalo, G. Tagliavini, F. Conti, D. Rossi, L. Benini, Xpulpnn: accelerating quantized neural networks on RISC-V processors through ISA extensions. In: 2020 Design, Automation Test in Europe Conference Exhibition (DATE), pp. 186–191 (2020). https://doi.org/10.23919/DATE48585.2020.9116529

  29. E. Wang, J.J. Davis, R. Zhao, H.C. Ng, X. Niu, W. Luk, P.Y.K. Cheung, G.A. Constantinides, Deep neural network approximation for custom hardware: where we’ve been, where we’re going. ACM Comput. Surv. 52, 2 (2019). https://doi.org/10.1145/3309551

    Article  Google Scholar 

  30. W. Jiang, L. Yang, S. Dasgupta, J. Hu, Y. Shi, Standing on the shoulders of giants: hardware and neural architecture co-search with hot start (2020). https://arxiv.org/abs/2007.09087

  31. P. Gysel, J. Pimentel, M. Motamedi, S. Ghiasi, Ristretto: a framework for empirical study of resource-efficient inference in convolutional neural networks. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5784–5789 (2018)

    Article  Google Scholar 

  32. K.O. Stanley, R. Miikkulainen, Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  33. E. Real, S. Moore, A. Selle, S. Saxena, Y.L. Suematsu, J. Tan, Q. Le, A. Kurakin, Large-Scale Evolution of Image Classifiers. arXiv e-prints arXiv:1703.01041 (2017)

  34. M. Suganuma, S. Shirakawa, T. Nagao, A genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’17, pp. 497–504. ACM (2017)

  35. Z. Lu, K. Deb, E.Goodman, W. Banzhaf, V.N. Boddeti, NSGANetV2: Evolutionary multi-objective surrogate-assisted neural architecture search. In: Computer Vision—ECCV 2020, pp. 35–51. Springer, Cham (2020)

  36. K. Deb, A. Pratap, S. Agarwal, T. Meyarivan, A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)

    Article  Google Scholar 

  37. Q. Lu, W. Jiang, X. Xu, Y. Shi, J. Hu, On neural architecture search for resource-constrained hardware platforms (2019). http://arxiv.org/abs/1911.00105

  38. Y. Lin, M. Yang, S. Han, NAAS: Neural accelerator architecture search. In: 2021 58th ACM/ESDA/IEEE Design Automation Conference (DAC) (2021)

  39. J.F. Miller, Cartesian Genetic Programming (Springer, Berlin, 2011)

    Book  Google Scholar 

  40. K. He, X. Zhang, S. Ren, J. Sun, Identity mappings in deep residual networks. In: Computer Vision—ECCV 2016, pp. 630–645. Springer (2016)

  41. T. Devries, G.W. Taylor, Improved regularization of convolutional neural networks with cutout. CoRR abs/1708.04552 (2017). http://arxiv.org/abs/1708.04552

  42. C. Shorten, T. Khoshgoftaar, A survey on image data augmentation for deep learning. J. Big Data 6, 1–48 (2019)

    Article  Google Scholar 

  43. , A. Krizhevsky, V. Nair, G. Hinton, CIFAR-10 (Canadian Institute for Advanced Research) http://www.cs.toronto.edu/~kriz/cifar.html

  44. Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A.Y. Ng, Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 (2011). http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf

  45. M. Fleischer, The measure of pareto optima applications to multi-objective metaheuristics, in Evolutionary Multi-criterion Optimization. (Springer, Berlin, 2003), pp. 519–533

    Chapter  Google Scholar 

  46. W. Jiang, L. Yang, E.H.M. Sha, Q. Zhuge, S. Gu, S. Dasgupta, Y. Shi, J. Hu, Hardware/software co-exploration of neural architectures. IEEE Trans. Comput.-Aid. Des. Integr. Circuits Syst. 39(12), 4805–4815 (2020). https://doi.org/10.1109/TCAD.2020.2986127

    Article  Google Scholar 

  47. M. Loni, S. Sinaei, A. Zoljodi, M. Daneshtalab, M. Sjödin, DeepMaker: a multi-objective optimization framework for deep neural networks in embedded systems. Microprocess. Microsyst. 73, 102989 (2020). https://doi.org/10.1016/j.micpro.2020.102989

    Article  Google Scholar 

  48. P. Achararit, M.A. Hanif, R.V.W. Putra, M. Shafique, Y. Hara-Azumi, APNAS: accuracy-and-performance-aware neural architecture search for neural hardware accelerators. IEEE Access 8, 165319–165334 (2020). https://doi.org/10.1109/ACCESS.2020.3022327

    Article  Google Scholar 

  49. Y. Jiang, X. Wang, W. Zhu, Hardware-aware transformable architecture search with efficient search space. In: 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6 (2020). https://doi.org/10.1109/ICME46284.2020.9102721

  50. T. Elsken, J.H. Metzen, F. Hutter, Efficient multi-objective neural architecture search via Lamarckian evolution. In: 7th International Conference on Learning Representations, ICLR 2019. OpenReview.net (2019)

  51. B. Zoph, V. Vasudevan, J. Shlens, Q.V. Le, Learning transferable architectures for scalable image recognition. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018). https://doi.org/10.1109/CVPR.2018.00907

  52. M. Wistuba, A. Rawat, T. Pedapati, A survey on neural architecture search. CoRR abs/1905.01392 (2019). http://arxiv.org/abs/1905.01392

  53. H. Tann, S. Hashemi, S. Reda, Lightweight Deep Neural Network Accelerators Using Approximate SW/HW Techniques (Springer, Berlin, 2019), pp. 289–305

    Google Scholar 

  54. H. Lee, E. Hyung, S.J. Hwang, Rapid neural architecture search by learning to generate graphs from datasets. In: International Conference on Learning Representations (2021). https://openreview.net/forum?id=rkQuFUmUOg3

  55. Z. Lu, G. Sreekumar, E. Goodman, W. Banzhaf, K. Deb, V.N. Boddeti, Neural architecture transfer. IEEE Trans. Pattern Anal. Mach. Intell. 43(9), 2971–2989 (2021). https://doi.org/10.1109/TPAMI.2021.3052758

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Czech Science Foundation Project 21-13001S. The computational experiments were supported by The Ministry of Education, Youth and Sports from the Large Infrastructures for Research, Experimental Development and Innovations project “e-Infrastructure CZ - LM2018140”.

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Brno University of Technology, Brno, Czech Republic

    Michal Pinos, Vojtech Mrazek & Lukas Sekanina

Authors
  1. Michal Pinos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vojtech Mrazek
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lukas Sekanina
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lukas Sekanina.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pinos, M., Mrazek, V. & Sekanina, L. Evolutionary approximation and neural architecture search. Genet Program Evolvable Mach 23, 351–374 (2022). https://doi.org/10.1007/s10710-022-09441-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10710-022-09441-z

Keywords

Search

Navigation