Skip to main content
SpringerLink
Log in

MTGP: Combining Metamorphic Testing and Genetic Programming

  • Conference paper
  • First Online:
Genetic Programming (EuroGP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13986))

Included in the following conference series:

Abstract

Genetic programming is an evolutionary approach known for its performance in program synthesis. However, it is not yet mature enough for a practical use in real-world software development, since usually many training cases are required to generate programs that generalize to unseen test cases. As in practice, the training cases have to be expensively hand-labeled by the user, we need an approach to check the program behavior with a lower number of training cases. Metamorphic testing needs no labeled input/output examples. Instead, the program is executed multiple times, first on a given (randomly generated) input, followed by related inputs to check whether certain user-defined relations between the observed outputs hold. In this work, we suggest MTGP, which combines metamorphic testing and genetic programming and study its performance and the generalizability of the generated programs. Further, we analyze how the generalizability depends on the number of given labeled training cases. We find that using metamorphic testing combined with labeled training cases leads to a higher generalization rate than the use of labeled training cases alone in almost all studied configurations. Consequently, we recommend researchers to use metamorphic testing in their systems if the labeling of the training data is expensive.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    More than one follow-up test is also possible, but in this work we focus on exactly one follow-up test.

References

  1. Aenugu, S., Spector, L.: Lexicase selection in learning classifier systems. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 356–364 (2019)

    Google Scholar 

  2. Arrieta, A.: Multi-objective metamorphic follow-up test case selection for deep learning systems. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1327–1335 (2022)

    Google Scholar 

  3. Błądek, I., Krawiec, K., Swan, J.: Counterexample-driven genetic programming: heuristic program synthesis from formal specifications. Evol. Comput. 26(3), 441–469 (2018)

    Article  Google Scholar 

  4. Chen, T.Y., et al.: Metamorphic testing: a review of challenges and opportunities. ACM Comput. Surv. (CSUR) 51(1), 1–27 (2018)

    Article  MathSciNet  Google Scholar 

  5. Chen, T.Y., Kuo, F.C., Liu, Y., Tang, A.: Metamorphic testing and testing with special values. In: SNPD, pp. 128–134 (2004)

    Google Scholar 

  6. Chen, T., Cheung, S., Yiu, S.: Metamorphic testing: a new approach for generating next test cases. Department of Computer Science, The Hong Kong University of Science and Technology, Technical report (1998)

    Google Scholar 

  7. Cramer, N.L.: A representation for the adaptive generation of simple sequential programs. In: Proceedings of the International Conference on Genetic Algorithms and the Applications, pp. 183–187 (1985)

    Google Scholar 

  8. Fagan, D., Fenton, M., O’Neill, M.: Exploring position independent initialisation in grammatical evolution. In: 2016 IEEE Congress on Evolutionary Computation (CEC), pp. 5060–5067. IEEE (2016)

    Google Scholar 

  9. Fenton, M., McDermott, J., Fagan, D., Forstenlechner, S., Hemberg, E., O’Neill, M.: PonyGE2: grammatical evolution in python. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1194–1201 (2017)

    Google Scholar 

  10. Forstenlechner, S., Fagan, D., Nicolau, M., O’Neill, M.: A grammar design pattern for arbitrary program synthesis problems in genetic programming. In: McDermott, J., Castelli, M., Sekanina, L., Haasdijk, E., García-Sánchez, P. (eds.) EuroGP 2017. LNCS, vol. 10196, pp. 262–277. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-55696-3_17

    Chapter  Google Scholar 

  11. Forstenlechner, S., Nicolau, M., Fagan, D., O’Neill, M.: Grammar design for derivation tree based genetic programming systems. In: Heywood, M.I., McDermott, J., Castelli, M., Costa, E., Sim, K. (eds.) EuroGP 2016. LNCS, vol. 9594, pp. 199–214. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30668-1_13

    Chapter  Google Scholar 

  12. Helmuth, T., Abdelhady, A.: Benchmarking parent selection for program synthesis by genetic programming. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 237–238 (2020)

    Google Scholar 

  13. Helmuth, T., Kelly, P.: PSB2: the second program synthesis benchmark suite. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 785–794 (2021)

    Google Scholar 

  14. Helmuth, T., McPhee, N.F., Pantridge, E., Spector, L.: Improving generalization of evolved programs through automatic simplification. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 937–944 (2017)

    Google Scholar 

  15. Helmuth, T., McPhee, N.F., Spector, L.: Lexicase selection for program synthesis: a diversity analysis. In: Riolo, R., Worzel, B., Kotanchek, M., Kordon, A. (eds.) Genetic Programming Theory and Practice XIII. GEC, pp. 151–167. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-34223-8_9

    Chapter  Google Scholar 

  16. Helmuth, T., McPhee, N.F., Spector, L.: Program synthesis using uniform mutation by addition and deletion. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1127–1134 (2018)

    Google Scholar 

  17. Helmuth, T., Pantridge, E., Spector, L.: On the importance of specialists for lexicase selection. Genet. Program. Evolvable Mach. 21(3), 349–373 (2020)

    Article  Google Scholar 

  18. Helmuth, T., Spector, L.: General program synthesis benchmark suite. In: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, pp. 1039–1046 (2015)

    Google Scholar 

  19. Helmuth, T., Spector, L.: Explaining and exploiting the advantages of down-sampled lexicase selection. In: ALIFE 2020: The 2020 Conference on Artificial Life, pp. 341–349. MIT Press (2020)

    Google Scholar 

  20. Hemberg, E., Kelly, J., O’Reilly, U.M.: On domain knowledge and novelty to improve program synthesis performance with grammatical evolution. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1039–1046 (2019)

    Google Scholar 

  21. Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection, vol. 1. MIT Press, Cambridge (1992)

    Google Scholar 

  22. Krawiec, K.: Behavioral Program Synthesis with Genetic Programming, vol. 618. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-27565-9

  23. Langdon, W.B., Krauss, O.: Evolving sqrt into 1/x via software data maintenance. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 1928–1936 (2020)

    Google Scholar 

  24. Schweim, D., Sobania, D., Rothlauf, F.: Effects of the training set size: A comparison of standard and down-sampled lexicase selection in program synthesis. In: 2022 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. IEEE (2022)

    Google Scholar 

  25. Sobania, D.: On the generalizability of programs synthesized by grammar-guided genetic programming. In: Hu, T., Lourenço, N., Medvet, E. (eds.) EuroGP 2021. LNCS, vol. 12691, pp. 130–145. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72812-0_9

    Chapter  Google Scholar 

  26. Sobania, D., Briesch, M., Rothlauf, F.: Choose your programming copilot: a comparison of the program synthesis performance of Github Copilot and genetic programming. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1019–1027 (2022)

    Google Scholar 

  27. Sobania, D., Rothlauf, F.: Challenges of program synthesis with grammatical evolution. In: Hu, T., Lourenço, N., Medvet, E., Divina, F. (eds.) EuroGP 2020. LNCS, vol. 12101, pp. 211–227. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-44094-7_14

    Chapter  Google Scholar 

  28. Sobania, D., Rothlauf, F.: A generalizability measure for program synthesis with genetic programming. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 822–829 (2021)

    Google Scholar 

  29. Sobania, D., Rothlauf, F.: Program synthesis with genetic programming: the influence of batch sizes. In: Medvet, E., Pappa, G., Xue, B. (eds.) Genetic Programming. EuroGP 2022. LNCS, vol. 13223, pp. 118–129. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-02056-8_8

  30. Sobania, D., Schweim, D., Rothlauf, F.: A comprehensive survey on program synthesis with evolutionary algorithms. IEEE Trans. Evol. Comput. (2022)

    Google Scholar 

  31. Spector, L.: Assessment of problem modality by differential performance of lexicase selection in genetic programming: a preliminary report. In: Proceedings of the 14th Annual Conference Companion on Genetic and Evolutionary Computation, pp. 401–408 (2012)

    Google Scholar 

  32. Spector, L., Klein, J., Keijzer, M.: The Push3 execution stack and the evolution of control. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation, pp. 1689–1696 (2005)

    Google Scholar 

  33. Spector, L., Robinson, A.: Genetic programming and autoconstructive evolution with the Push programming language. Genet. Program. Evolvable Mach. 3(1), 7–40 (2002)

    Article  MATH  Google Scholar 

  34. Whigham, P.A., et al.: Grammatically-based genetic programming. In: Proceedings of the Workshop on Genetic Programming: From Theory to Real-world Applications, vol. 16, pp. 33–41. Citeseer (1995)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Johannes Gutenberg University, Mainz, Germany

    Dominik Sobania, Martin Briesch, Philipp Röchner & Franz Rothlauf

Authors
  1. Dominik Sobania
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martin Briesch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Philipp Röchner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Franz Rothlauf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dominik Sobania .

Editor information

Editors and Affiliations

  1. Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

    Gisele Pappa

  2. Università degli studi di Torino, Turin, Italy

    Mario Giacobini

  3. Brno University of Technology, Brno, Czech Republic

    Zdenek Vasicek

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sobania, D., Briesch, M., Röchner, P., Rothlauf, F. (2023). MTGP: Combining Metamorphic Testing and Genetic Programming. In: Pappa, G., Giacobini, M., Vasicek, Z. (eds) Genetic Programming. EuroGP 2023. Lecture Notes in Computer Science, vol 13986. Springer, Cham. https://doi.org/10.1007/978-3-031-29573-7_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-29573-7_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-29572-0

  • Online ISBN: 978-3-031-29573-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation