Skip to main content
SpringerLink
Log in

Improving Generalization of Evolutionary Feature Construction with Minimal Complexity Knee Points in Regression

  • Conference paper
  • First Online:
Genetic Programming (EuroGP 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14631))

Included in the following conference series:

  • 119 Accesses

Abstract

Genetic programming-based evolutionary feature construction is a widely used technique for automatically enhancing the performance of a regression algorithm. While it has achieved great success, a challenging problem in feature construction is the issue of overfitting, which has led to the development of many multi-objective methods to control overfitting. However, for multi-objective methods, a key issue is how to select the final model from the front with different trade-offs. To address this challenge, in this paper, we propose a novel minimal complexity knee point selection strategy in evolutionary multi-objective feature construction for regression to select the final model for making predictions. Experimental results on 58 datasets demonstrate the effectiveness and competitiveness of this strategy when compared to eight existing methods. Furthermore, an ensemble of the proposed strategy and existing model selection strategies achieves the best performance and outperforms four popular machine learning algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Source code: https://anonymous.4open.science/r/Knee-GP/.

References

  1. Agapitos, A., Loughran, R., Nicolau, M., Lucas, S., O’Neill, M., Brabazon, A.: A survey of statistical machine learning elements in genetic programming. IEEE Trans. Evol. Comput. 23(6), 1029–1048 (2019)

    Article  Google Scholar 

  2. Al-Sahaf, H., et al.: A survey on evolutionary machine learning. J. R. Soc. N. Z. 49(2), 205–228 (2019)

    Article  Google Scholar 

  3. Banzhaf, W., Nordin, P., Keller, R.E., Francone, F.D.: Genetic Programming: An Introduction: On the Automatic Evolution of Computer Programs and Its Applications. Morgan Kaufmann Publishers Inc. (1998)

    Google Scholar 

  4. Bi, Y., Xue, B., Zhang, M.: Dual-tree genetic programming for few-shot image classification. IEEE Trans. Evol. Comput. 26(3), 555–569 (2021)

    Article  Google Scholar 

  5. Bi, Y., Xue, B., Zhang, M.: Learning and sharing: a multitask genetic programming approach to image feature learning. IEEE Trans. Evol. Comput. 26(2), 218–232 (2021)

    Article  Google Scholar 

  6. Branke, J., Deb, K., Dierolf, H., Osswald, M.: Finding knees in multi-objective optimization. In: Yao, X., et al. (eds.) PPSN 2004. LNCS, vol. 3242, pp. 722–731. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30217-9_73

    Chapter  Google Scholar 

  7. Chaudhuri, S., Deb, K.: An interactive evolutionary multi-objective optimization and decision making procedure. Appl. Soft Comput. 10(2), 496–511 (2010)

    Article  Google Scholar 

  8. Chen, Q., Xue, B., Zhang, M.: Rademacher complexity for enhancing the generalization of genetic programming for symbolic regression. IEEE Trans. Cybern. 52(4), 2382–2395 (2022)

    Article  Google Scholar 

  9. Chen, Q., Zhang, M., Xue, B.: Feature selection to improve generalization of genetic programming for high-dimensional symbolic regression. IEEE Trans. Evol. Comput. 21(5), 792–806 (2017)

    Article  Google Scholar 

  10. Chen, Q., Zhang, M., Xue, B.: Structural risk minimization-driven genetic programming for enhancing generalization in symbolic regression. IEEE Trans. Evol. Comput. 23(4), 703–717 (2018)

    Article  Google Scholar 

  11. Deb, K., Gupta, S.: Understanding knee points in bicriteria problems and their implications as preferred solution principles. Eng. Optim. 43(11), 1175–1204 (2011)

    Article  MathSciNet  Google Scholar 

  12. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)

    Article  Google Scholar 

  13. de França, F.O.: Transformation-interaction-rational representation for symbolic regression: a detailed analysis of SRBench results. ACM Trans. Evol. Learn. (2023)

    Google Scholar 

  14. de Franca, F., et al.: Interpretable symbolic regression for data science: analysis of the 2022 competition. arXiv preprint arXiv:2304.01117 (2023)

  15. Gaier, A., Ha, D.: Weight agnostic neural networks. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  16. Gonçalves, I., Silva, S.: Balancing learning and overfitting in genetic programming with interleaved sampling of training data. In: Krawiec, K., Moraglio, A., Hu, T., Etaner-Uyar, A.Ş, Hu, B. (eds.) EuroGP 2013. LNCS, vol. 7831, pp. 73–84. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37207-0_7

    Chapter  Google Scholar 

  17. La Cava, W., Moore, J.H.: Learning feature spaces for regression with genetic programming. Genet. Program Evolvable Mach. 21, 433–467 (2020)

    Article  Google Scholar 

  18. La Cava, W., Silva, S., Danai, K., Spector, L., Vanneschi, L., Moore, J.H.: Multidimensional genetic programming for multiclass classification. Swarm Evol. Comput. 44, 260–272 (2019)

    Article  Google Scholar 

  19. Li, K., Nie, H., Gao, H., Yao, X.: Posterior decision making based on decomposition-driven knee point identification. IEEE Trans. Evol. Comput. 26(6), 1409–1423 (2021)

    Article  Google Scholar 

  20. Muñoz, L., Trujillo, L., Silva, S., Castelli, M., Vanneschi, L.: Evolving multidimensional transformations for symbolic regression with M3GP. Memetic Comput. 11, 111–126 (2019)

    Article  Google Scholar 

  21. Muñoz, M.A., et al.: An instance space analysis of regression problems. ACM Trans. Knowl. Discov. Data (TKDD) 15(2), 1–25 (2021)

    Article  Google Scholar 

  22. Neshatian, K., Zhang, M., Andreae, P.: A filter approach to multiple feature construction for symbolic learning classifiers using genetic programming. IEEE Trans. Evol. Comput. 16(5), 645–661 (2012)

    Article  Google Scholar 

  23. Ni, J., Drieberg, R.H., Rockett, P.I.: The use of an analytic quotient operator in genetic programming. IEEE Trans. Evol. Comput. 17(1), 146–152 (2012)

    Article  Google Scholar 

  24. Ni, J., Rockett, P.: Tikhonov regularization as a complexity measure in multiobjective genetic programming. IEEE Trans. Evol. Comput. 19(2), 157–166 (2014)

    Article  Google Scholar 

  25. Nicolau, M., Agapitos, A.: Choosing function sets with better generalisation performance for symbolic regression models. Genet. Program Evolvable Mach. 22(1), 73–100 (2021)

    Article  Google Scholar 

  26. Olson, R.S., La Cava, W., Orzechowski, P., Urbanowicz, R.J., Moore, J.H.: PMLB: a large benchmark suite for machine learning evaluation and comparison. BioData Min. 10, 1–13 (2017)

    Article  Google Scholar 

  27. Orzechowski, P., La Cava, W., Moore, J.H.: Where are we now? A large benchmark study of recent symbolic regression methods. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1183–1190 (2018)

    Google Scholar 

  28. Peng, B., Wan, S., Bi, Y., Xue, B., Zhang, M.: Automatic feature extraction and construction using genetic programming for rotating machinery fault diagnosis. IEEE Trans. Cybern. 51(10), 4909–4923 (2020)

    Article  Google Scholar 

  29. Rachmawati, L., Srinivasan, D.: Multiobjective evolutionary algorithm with controllable focus on the knees of the pareto front. IEEE Trans. Evol. Comput. 13(4), 810–824 (2009)

    Article  Google Scholar 

  30. Ramirez-Atencia, C., Mostaghim, S., Camacho, D.: A knee point based evolutionary multi-objective optimization for mission planning problems. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1216–1223 (2017)

    Google Scholar 

  31. Schütze, O., Laumanns, M., Coello, C.A.C.: Approximating the knee of an MOP with stochastic search algorithms. In: Rudolph, G., Jansen, T., Beume, N., Lucas, S., Poloni, C. (eds.) PPSN 2008. LNCS, vol. 5199, pp. 795–804. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-87700-4_79

    Chapter  Google Scholar 

  32. Telikani, A., Tahmassebi, A., Banzhaf, W., Gandomi, A.H.: Evolutionary machine learning: a survey. ACM Comput. Surv. (CSUR) 54(8), 1–35 (2021)

    Article  Google Scholar 

  33. Vanneschi, L., Castelli, M.: Soft target and functional complexity reduction: a hybrid regularization method for genetic programming. Expert Syst. Appl. 177, 114929 (2021)

    Article  Google Scholar 

  34. Vanneschi, L., Castelli, M., Silva, S.: Measuring bloat, overfitting and functional complexity in genetic programming. In: Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, pp. 877–884 (2010)

    Google Scholar 

  35. Virgolin, M., Alderliesten, T., Bosman, P.A.: On explaining machine learning models by evolving crucial and compact features. Swarm Evol. Comput. 53, 100640 (2020)

    Article  Google Scholar 

  36. Zhang, B.T., Muhlenbein, H., et al.: Evolving optimal neural networks using genetic algorithms with occam’s razor. Complex Syst. 7(3), 199–220 (1993)

    Google Scholar 

  37. Zhang, F., Mei, Y., Nguyen, S., Zhang, M.: Evolving scheduling heuristics via genetic programming with feature selection in dynamic flexible job-shop scheduling. IEEE Trans. Cybern. 51(4), 1797–1811 (2020)

    Article  Google Scholar 

  38. Zhang, F., Mei, Y., Nguyen, S., Zhang, M.: Collaborative multifidelity-based surrogate models for genetic programming in dynamic flexible job shop scheduling. IEEE Trans. Cybern. 52(8), 8142–8156 (2021)

    Article  Google Scholar 

  39. Zhang, H., Chen, Q., Xue, B., Banzhaf, W., Zhang, M.: Modular multi-tree genetic programming for evolutionary feature construction for regression. IEEE Trans. Evol. Comput. (2023)

    Google Scholar 

  40. Zhang, H., Chen, Q., Xue, B., Banzhaf, W., Zhang, M.: A semantic-based hoist mutation operator for evolutionary feature construction in regression. IEEE Trans. Evol. Comput. (2023). https://doi.org/10.1109/TEVC.2023.3331234

  41. Zhang, H., Zhou, A., Chen, Q., Xue, B., Zhang, M.: SR-Forest: a genetic programming based heterogeneous ensemble learning method. IEEE Trans. Evol. Comput. (2023)

    Google Scholar 

  42. Zhang, H., Zhou, A., Qian, H., Zhang, H.: PS-tree: a piecewise symbolic regression tree. Swarm Evol. Comput. 71, 101061 (2022)

    Article  Google Scholar 

  43. Zhang, H., Zhou, A., Zhang, H.: An evolutionary forest for regression. IEEE Trans. Evol. Comput. 26(4), 735–749 (2021)

    Article  Google Scholar 

  44. Zhang, X., Tian, Y., Jin, Y.: A knee point-driven evolutionary algorithm for many-objective optimization. IEEE Trans. Evol. Comput. 19(6), 761–776 (2014)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Data Science and Artificial Intelligence and School of Engineering and Computer Science, Victoria University of Wellington, PO Box 600, Wellington, 6140, New Zealand

    Hengzhe Zhang, Qi Chen, Bing Xue & Mengjie Zhang

  2. Department of Computer Science and Engineering in the College of Engineering and BEACON Center, Michigan State University, East Lansing, MI, 48824, USA

    Wolfgang Banzhaf

Authors
  1. Hengzhe Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bing Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wolfgang Banzhaf
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mengjie Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hengzhe Zhang .

Editor information

Editors and Affiliations

  1. University of Torino, Grugliasco, Italy

    Mario Giacobini

  2. Victoria University of Wellington, Wellington, New Zealand

    Bing Xue

  3. University of Trieste, Trieste, Italy

    Luca Manzoni

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, H., Chen, Q., Xue, B., Banzhaf, W., Zhang, M. (2024). Improving Generalization of Evolutionary Feature Construction with Minimal Complexity Knee Points in Regression. In: Giacobini, M., Xue, B., Manzoni, L. (eds) Genetic Programming. EuroGP 2024. Lecture Notes in Computer Science, vol 14631. Springer, Cham. https://doi.org/10.1007/978-3-031-56957-9_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56957-9_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56956-2

  • Online ISBN: 978-3-031-56957-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation