Skip to main content
SpringerLink
Log in

Small Solutions for Real-World Symbolic Regression Using Denoising Autoencoder Genetic Programming

  • Conference paper
  • First Online:
Genetic Programming (EuroGP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13986))

Included in the following conference series:

Abstract

Denoising Autoencoder Genetic Programming (DAE-GP) is a model-based evolutionary algorithm that uses denoising autoencoder long short-term memory networks as probabilistic model to replace the standard recombination and mutation operators of genetic programming (GP). In this paper, we use the DAE-GP to solve a set of nine standard real-world symbolic regression tasks. We compare the prediction quality of the DAE-GP to standard GP, geometric semantic GP (GSGP), and the gene-pool optimal mixing evolutionary algorithm for GP (GOMEA-GP), and find that the DAE-GP shows similar prediction quality using a much lower number of fitness evaluations than GSGP or GOMEA-GP. In addition, the DAE-GP consistently finds small solutions. The best candidate solutions of the DAE-GP are 69% smaller (median number of nodes) than the best candidate solutions found by standard GP. An analysis of the bias of the selection and variation step for both the DAE-GP and standard GP gives insight into why differences in solution size exist: the strong increase in solution size for standard GP is a result of both selection and variation bias. The results highlight that learning and sampling from a probabilistic model is a promising alternative to classic GP variation operators where the DAE-GP is able to generate small solutions for real-world symbolic regression tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For a detailed description on DAE-LSTM, refer to Wittenberg [31].

References

  1. Brooks, T.F., Pope, D.S., Marcolini, M.A.: Airfoil self-noise and prediction, vol. 1218. In: National Aeronautics and Space Administration, Office of Management, Scientific and Technical Information Division (1989)

    Google Scholar 

  2. Chollet, F.: keras (2015). https://github.com/fchollet/keras

  3. Cortez, P., Cerdeira, A., Almeida, F., Matos, T., Reis, J.: Modeling wine preferences by data mining from physicochemical properties. Decis. Support Syst. 47(4), 547–553 (2009)

    Article  Google Scholar 

  4. Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml

  5. Fortin, F.A., De Rainville, F.M., Gardner, M.A., Parizeau, M., Gagńe, C.: DEAP: evolutionary algorithms made easy. J. Mach. Learn. Res. 13(1), 2171–2175 (2012)

    MathSciNet  Google Scholar 

  6. Gerritsma, J., Onnink, R., Versluis, A.: Geometry, resistance and stability of the delft systematic yacht hull series. Int. Shipbuild. Prog. 28(328), 276–297 (1981)

    Article  Google Scholar 

  7. Harrison, D., Jr., Rubinfeld, D.L.: Hedonic housing prices and the demand for clean air. J. Environ. Econ. Manag. 5(1), 81–102 (1978)

    Article  MATH  Google Scholar 

  8. Hasegawa, Y., Iba, H.: Estimation of Bayesian network for program generation. In: Proceedings of the Third Asian-Pacific Workshop on Genetic Programming, pp. 35–46. Hanoi, Vietnam (2006)

    Google Scholar 

  9. Hasegawa, Y., Iba, H.: Estimation of distribution algorithm based on probabilistic grammar with latent annotations. In: Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2007), pp. 1043–1050. IEEE (2007). https://doi.org/10.1109/CEC.2007.4424585

  10. Hasegawa, Y., Iba, H.: A Bayesian network approach to program generation. IEEE Trans. Evol. Comput. 12(6), 750–764 (2008). https://doi.org/10.1109/tevc.2008.915999

    Article  Google Scholar 

  11. Kim, K., Shan, Y., Nguyen, X.H., McKay, R.I.: Probabilistic model building in genetic programming: a critical review. Genet. Program Evolvable Mach. 15(2), 115–167 (2014). https://doi.org/10.1007/s10710-013-9205-x

    Article  Google Scholar 

  12. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations. San Diego, CA, USA (2015)

    Google Scholar 

  13. Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge, London (1992)

    MATH  Google Scholar 

  14. La Cava, W., Spector, L., Danai, K.: Epsilon-lexicase selection for regression. In: Proceedings of the Genetic and Evolutionary Computation Conference 2016 (GECCO 2016), pp. 741–748. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2908812.2908898

  15. Martins, J.F.B.S., Oliveira, L.O.V.B., Miranda, L.F., Casadei, F., Pappa, G.L.: Solving the exponential growth of symbolic regression trees in geometric semantic genetic programming. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2018), pp. 1151–1158. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3205455.3205593

  16. de Melo, V.V., Vargas, D.V., Banzhaf, W.: Batch tournament selection for genetic programming: the quality of lexicase, the speed of tournament. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019), pp. 994–1002. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3321707.3321793

  17. Moraglio, A., Krawiec, K., Johnson, C.G.: Geometric semantic genetic programming. In: Coello, C.A.C., Cutello, V., Deb, K., Forrest, S., Nicosia, G., Pavone, M. (eds.) PPSN 2012. LNCS, vol. 7491, pp. 21–31. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32937-1_3

    Chapter  Google Scholar 

  18. Ni, J., Drieberg, R.H., Rockett, P.I.: The use of an analytic quotient operator in genetic programming. IEEE Trans. Evol. Comput. 17(1), 146–152 (2013). https://doi.org/10.1109/TEVC.2012.2195319

    Article  Google Scholar 

  19. Orzechowski, P., La Cava, W., Moore, J.H.: Where are we now? A large benchmark study of recent symbolic regression methods. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2018), pp. 1183–1190. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3205455.3205539

  20. Probst, M., Rothlauf, F.: Harmless overfitting: using denoising autoencoders in estimation of distribution algorithms. J. Mach. Learn. Res. 21(78), 1–31 (2020). http://jmlr.org/papers/v21/16-543.html

    MathSciNet  MATH  Google Scholar 

  21. Ratle, A., Sebag, M.: Avoiding the bloat with stochastic grammar-based genetic programming. In: Collet, P., Fonlupt, C., Hao, J.-K., Lutton, E., Schoenauer, M. (eds.) EA 2001. LNCS, vol. 2310, pp. 255–266. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-46033-0_21

    Chapter  Google Scholar 

  22. Rothlauf, F.: Design of Modern Heuristics: Principles and Application, 1st edn. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-540-72962-4

  23. Salustowicz, R., Schmidhuber, J.: Probabilistic incremental program evolution. Evol. Comput. 5(2), 123–141 (1997). https://doi.org/10.1162/evco.1997.5.2.123

    Article  Google Scholar 

  24. Schmidt, M., Lipson, H.: Age-fitness pareto optimization. In: Riolo, R., McConaghy, T., Vladislavleva, E. (eds.) Genetic Programming Theory and Practice VIII, pp. 129–146. Springer, New York (2011). https://doi.org/10.1007/978-1-4419-7747-2_8

  25. Srivastava, N., Mansimov, E., Salakhutdinov, R.: Unsupervised learning of video representations using LSTMs. In: Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), pp. 843–852. ACM, Lille, France (2015). https://doi.org/10.5555/3045118.3045209

  26. Tsanas, A., Little, M.A., McSharry, P.E., Ramig, L.O.: Accurate telemonitoring of Parkinson’s disease progression by noninvasive speech tests. IEEE Trans. Biomed. Eng. 57(4), 884–893 (2009)

    Article  Google Scholar 

  27. Tsanas, A., Xifara, A.: Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy Build. 49, 560–567 (2012)

    Article  Google Scholar 

  28. Vaswani, A., et al.: Attention is all you need. Adv. Neural. Inf. Process. Syst. 30, 5998–6008 (2017)

    Google Scholar 

  29. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning (ICML 2008), pp. 1096–1103. ACM, Helsinki, Finland (2008). https://doi.org/10.1145/1390156.1390294

  30. Virgolin, M., Alderliesten, T., Witteveen, C., Bosman, P.A.N.: Improving model-based genetic programming for symbolic regression of small expressions. Evol. Comput. 29(2), 211–237 (2021). https://doi.org/10.1162/evco_a_00278

    Article  Google Scholar 

  31. Wittenberg, D.: Using denoising autoencoder genetic programming to control exploration and exploitation in search. In: Medvet, E., Pappa, G., Xue, B. (eds.) Genetic Programming (EuroGP 2022). LNCS, vol. 13223, pp. 102–117. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-02056-8_7

  32. Wittenberg, D., Rothlauf, F.: Denoising autoencoder genetic programming for real-world symbolic regression. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO 2022), pp. 612–614. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3520304.3528921

  33. Wittenberg, D., Rothlauf, F., Schweim, D.: DAE-GP: denoising autoencoder LSTM networks as probabilistic models in estimation of distribution genetic programming. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference (GECCO 2020), pp. 1037–1045. ACM, New York, NY, USA (2020). https://doi.org/10.1145/3377930.3390180

  34. Wong, P.K., Lo, L.Y., Wong, M.L., Leung, K.S.: Grammar-based genetic programming with Bayesian network. In: IEEE Congress on Evolutionary Computation (CEC 2014), pp. 739–746. IEEE, Beijing, China (2014)

    Google Scholar 

  35. Wong, P.K., Lo, L.Y., Wong, M.L., Leung, K.S.: Grammar-based genetic programming with dependence learning and Bayesian network classifier. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2014), pp. 959–966. ACM, Vancouver, Canada (2014). https://doi.org/10.1145/2576768.2598256

  36. Yanai, K., Iba, H.: Estimation of distribution programming based on Bayesian network. In: IEEE Congress on Evolutionary Computation (CEC 2003), pp. 1618–1625. IEEE, Canberra, Australia (2003). https://doi.org/10.1109/CEC.2003.1299866

  37. Yeh, I.C.: Modeling of strength of high-performance concrete using artificial neural networks. Cem. Concr. Res. 28(12), 1797–1808 (1998)

    Article  Google Scholar 

Download references

Acknowledgements

We thank our group in Mainz for previous work and insightful discussions on this topic. Parts of this research were conducted using the supercomputer Mogon offered by Johannes Gutenberg University Mainz (hpc.uni-mainz.de), which is a member of the AHRP (Alliance for High Performance Computing in Rhineland Palatinate, www.ahrp.info) and the Gauss Alliance e.V. The authors gratefully acknowledge the computing time granted on the supercomputer Mogon at Johannes Gutenberg University Mainz (hpc.uni-mainz.de).

Author information

Authors and Affiliations

  1. Johannes Gutenberg University, Mainz, Germany

    David Wittenberg & Franz Rothlauf

Authors
  1. David Wittenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Franz Rothlauf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Wittenberg .

Editor information

Editors and Affiliations

  1. Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

    Gisele Pappa

  2. Università degli studi di Torino, Turin, Italy

    Mario Giacobini

  3. Brno University of Technology, Brno, Czech Republic

    Zdenek Vasicek

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wittenberg, D., Rothlauf, F. (2023). Small Solutions for Real-World Symbolic Regression Using Denoising Autoencoder Genetic Programming. In: Pappa, G., Giacobini, M., Vasicek, Z. (eds) Genetic Programming. EuroGP 2023. Lecture Notes in Computer Science, vol 13986. Springer, Cham. https://doi.org/10.1007/978-3-031-29573-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-29573-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-29572-0

  • Online ISBN: 978-3-031-29573-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation