Skip to main content
SpringerLink
Log in

Using genetic programming to evolve action selection rules in traversal-based automated software testing: results obtained with the TESTAR tool

  • Regular Research Paper
  • Published:
Memetic Computing Aims and scope Submit manuscript

Abstract

Traversal-based automated software testing involves testing an application via its graphical user interface (GUI) and thereby taking the user’s point of view and executing actions in a human-like manner. These actions are decided on the fly, as the software under test (SUT) is being run, as opposed to being set up in the form of a sequence prior to the testing, a sequence that is then used to exercise the SUT. In practice, random choice is commonly used to decide which action to execute at each state (a procedure commonly referred to as monkey testing), but a number of alternative mechanisms have also been proposed in the literature. Here we propose using genetic programming (GP) to evolve such an action selection strategy, defined as a list of IF-THEN rules. Genetic programming has proved to be suited for evolving all sorts of programs, and rules in particular, provided adequate primitives (functions and terminals) are defined. These primitives must aim to extract the most relevant information from the SUT and the dynamics of the testing process. We introduce a number of such primitives suited to the problem at hand and evaluate their usefulness based on various metrics. We carry out experiments and compare the results with those obtained by random selection and also by Q-learning, a reinforcement learning technique. Three applications are used as Software Under Test (SUT) in the experiments. The analysis shows the potential of GP to evolve action selection strategies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://testar.org.

  2. E.g. display position, widget size, ancestor widgets, etc.

  3. Note that ascertaining whether these failures are associated to any defects is beyond the scope of the TESTAR tool.

References

  1. Aho P, Menz N, Rty T (2013) Dynamic reverse engineering of GUI models for testing. In: Proceedings of 2013 international conference on control, decision and information technologies (CoDIT’13)

  2. Aho P, Oliveira R, Algroth E, Vos T (2016) Evolution of automated testing of software systems through graphical user interface. In: Procs. of the 1st international conference on advances in computation, communications and services (ACCSE 2016), Valencia, pp 16–21

  3. Alegroth E, Feldt R, Ryrholm L (2014) Visual GUI testing in practice: challenges, problems and limitations. Empir Softw Eng 20:694–744. https://doi.org/10.1007/s10664-013-9293-5

    Article  Google Scholar 

  4. Barr ET, Harman M, McMinn P, Shahbaz M, Yoo S (2015) The oracle problem in software testing: a survey. IEEE Trans Softw Eng 41(5):507–525

    Article  Google Scholar 

  5. Bauersfeld S, Vos TEJ (2012) A reinforcement learning approach to automated GUI robustness testing. In: Fast abstracts of the 4th symposium on search-based software engineering (SSBSE 2012), pp 7–12

  6. Bauersfeld S, de Rojas A, Vos T (2014) Evaluating rogue user testing in industry: an experience report. In: 2014 IEEE eighth international conference on research challenges in information science (RCIS), pp 1–10. https://doi.org/10.1109/RCIS.2014.6861051

  7. Bauersfeld S, Vos TEJ, Condori-Fernández N, Bagnato A, Brosse E (2014) Evaluating the TESTAR tool in an industrial case study. In: 2014 ACM-IEEE international symposium on empirical software engineering and measurement, ESEM 2014, Torino, Italy, September 18–19, 2014, p 4

  8. Bauersfeld S, Wappler S, Wegener J (2011) A metaheuristic approach to test sequence generation for applications with a GUI. In: Cohen MB, Ó Cinnéide M (eds) Search based software engineering: third international symposium, SSBSE 2011, Szeged, Hungary, September 10-12, 2011. Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 173–187

  9. Brameier MF, Banzhaf W (2010) Linear genetic programming, 1st edn. Springer, New York

    MATH  Google Scholar 

  10. Chaudhary N, Sangwan O (2016) Metrics for event driven software. Int J Adv Comput Sci Appl 7(1):85–89

    Google Scholar 

  11. Esparcia-Alcázar AI, Almenar F, Martínez M, Rueda U, Vos TE (2016) Q-learning strategies for action selection in the TESTAR automated testing tool. In: Proceedings of META 2016 6th international conference on metaheuristics and nature inspired computing, pp 174–180

  12. Esparcia-Alcázar AI, Almenar F, Rueda U, Vos TEJ (2017) Evolving rules for action selection in automated testing via genetic programming–a first approach. In: Squillero G, Sim K (eds) Applications of evolutionary computation: 20th European conference, evoapplications 2017, Amsterdam, The Netherlands, April 19–21, 2017, Proceedings, part II. Springer, pp 82–95. https://doi.org/10.1007/978-3-319-55792-2_6

  13. Esparcia-Alcázar AI, Moravec J (2013) Fitness approximation for bot evolution in genetic programming. Soft Comput 17(8):1479–1487. https://doi.org/10.1007/s00500-012-0965-7

    Article  Google Scholar 

  14. He W, Zhao R, Zhu Q (2015) Integrating evolutionary testing with reinforcement learning for automated test generation of object-oriented software. Chin J Electron 24(1):38–45

    Article  Google Scholar 

  15. Koza JR (1992) Genetic programming: on the programming of computers by means of natural selection. MIT Press, Cambridge

    MATH  Google Scholar 

  16. Lehman J, Stanley KO (2011) Novelty search and the problem with objectives. In: Riolo R, Vladislavleva E, Moore JH (eds) Genetic programming theory and practice IX, genetic and evolutionary computation. Springer, New York, pp 37–56

    Google Scholar 

  17. Memon AM, Soffa ML, Pollack ME (2001) Coverage criteria for GUI testing. In: Proceedings of ESEC/FSE 2001, pp 256–267

  18. Rueda U, Vos TEJ, Almenar F, Martínez MO, Esparcia-Alcázar AI (2015) TESTAR: from academic prototype towards an industry-ready tool for automated testing at the user interface level. In: Canos JH, Gonzalez Harbour M (eds) Actas de las XX Jornadas de Ingeniería del Software y Bases de Datos (JISBD 2015), pp 236–245

  19. Seesing A, Gross HG (2006) A genetic programming approach to automated test generation for object-oriented software. Int Trans Syst Sci Appl 1(2):127–134

    Google Scholar 

  20. Vos TE, Kruse PM, Condori-Fernández N, Bauersfeld S, Wegener J (2015) TESTAR: tool support for test automation at the user interface level. Int J Inf Syst Model Des 6(3):46–83. https://doi.org/10.4018/IJISMD.2015070103

    Article  Google Scholar 

  21. Wappler S, Wegener J (2006) Evolutionary unit testing of object-oriented software using strongly-typed genetic programming. In: Proceedings of the 8th annual conference on genetic and evolutionary computation, GECCO’06. ACM, New York, NY, USA, pp 1925–1932. URL https://doi.org/10.1145/1143997.1144317

  22. Watkins C (1989) Learning from delayed rewards. Ph.D. Thesis. Cambridge University

Download references

Acknowledgements

The authors wish to thank Xara Sharman for her help with the graphics.

Author information

Authors and Affiliations

  1. Department of Systems Engineering and Control, Universitat Politècnica de València, Valencia, Spain

    Anna I. Esparcia-Alcázar

  2. Software Production Methods Research Centre, Universitat Politècnica de València, Valencia, Spain

    Anna I. Esparcia-Alcázar, Francisco Almenar, Tanja E. J. Vos & Urko Rueda

  3. Open Universiteit, Heerlen, The Netherlands

    Tanja E. J. Vos

  4. Etra I+D, Valencia, Spain

    Francisco Almenar

Authors
  1. Anna I. Esparcia-Alcázar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Francisco Almenar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tanja E. J. Vos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Urko Rueda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anna I. Esparcia-Alcázar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Esparcia-Alcázar, A.I., Almenar, F., Vos, T.E.J. et al. Using genetic programming to evolve action selection rules in traversal-based automated software testing: results obtained with the TESTAR tool. Memetic Comp. 10, 257–265 (2018). https://doi.org/10.1007/s12293-018-0263-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12293-018-0263-8

Keywords

Search

Navigation