Test Case Evaluation and Input Domain Reduction strategies for the Evolutionary Testing of Object-Oriented software

https://doi.org/10.1016/j.infsof.2009.06.009Get rights and content

Abstract

In Evolutionary Testing, meta-heuristic search techniques are used for generating test data. The focus of our research is on employing evolutionary algorithms for the structural unit-testing of Object-Oriented programs. Relevant contributions include the introduction of novel methodologies for automation, search guidance and Input Domain Reduction; the strategies proposed were empirically evaluated with encouraging results.

Test cases are evolved using the Strongly-Typed Genetic Programming technique. Test data quality evaluation includes instrumenting the test object, executing it with the generated test cases, and tracing the structures traversed in order to derive coverage metrics. The methodology for efficiently guiding the search process towards achieving full structural coverage involves favouring test cases that exercise problematic structures. Purity Analysis is employed as a systematic strategy for reducing the search space.

Introduction

Software testing is expensive, typically consuming roughly half of the total costs involved in software development while adding nothing to the raw functionality of the final product. Yet, it remains the primary method through which confidence in software is achieved [6]. A large amount of the resources spent on testing are applied on the difficult and time consuming task of locating quality test data; automating this process is vital to advance the state-of-the-art in software testing. However, automation in this area has been quite limited, mainly because the exhaustive enumeration of a program’s input is unfeasible for any reasonably-sized program, and random methods are unlikely to exercise “deeper” features of software [25].

Meta-heuristic search techniques, like Evolutionary Algorithms – high-level frameworks which utilise heuristics, inspired by genetics and natural selection, in order to find solutions to combinatorial problems at a reasonable computational cost [4] – are natural candidates to address this problem, since the input space is typically large but well defined, and test goal can usually be expressed as a fitness function [10].

The application of Evolutionary Algorithms to test data generation is often referred to as Evolutionary Testing [39] or Search-Based Testing [25]. Approaches have been proposed that focus on the usage of Genetic Algorithms [16], [17], [39], [48], Ant Colony Optimization [22], Genetic Programming [38], Strongly-Typed Genetic Programming [43], [45], and Memetic Algorithms [1].

Evolutionary Testing is an emerging methodology for automatically generating high quality test data. It is, however, a difficult subject, especially if the aim is to implement an automated solution, viable with a reasonable amount of computational effort, which is adaptable to a wide range of test objects. Significant success has been achieved by applying this technique to the automatic generation of unit-test cases for procedural software [24], [25]. The application of search-based strategies for Object-Oriented unit-testing is, however, fairly recent [39] and is yet to be investigated comprehensively [11].

The focus of our research is precisely on developing a solution for employing Evolutionary Algorithms for generating test sets for the structural unit-testing of Object-Oriented programs. Our approach involves representing and evolving test cases using the Strongly-Typed Genetic Programming technique [28]. The methodology for evaluating the quality of test cases includes instrumenting the program under test, and executing it using the generated test cases as inputs with the intention of collecting trace information with which to derive coverage metrics. The aim is that of efficiently guiding the search process towards achieving full structural coverage of the program under test. These concepts have been implemented into the eCrash automated test case generation tool – which will be described below.

Our main goals are those of defining strategies for addressing the challenges posed by the Object-Oriented paradigm and of proposing methodologies for enhancing the efficiency of search-based testing approaches. The primary contributions of this work are the following:

  • Presenting a strategy for Test Case Evaluation and search guidance, which involves allowing unfeasible test cases (i.e., those that terminate prematurely due to a runtime exception) to be considered at certain stages of the evolutionary search – namely, once the feasible test cases that are being bred cease to be interesting.

  • Introducing a novel Input Domain Reduction methodology, based on the concept of Purity Analysis, which allows the identification and removal of entries that are irrelevant to the search problem because they do not contribute to the definition of test scenarios.

Additionally, our methodology for automated test case generation is thoroughly described and validated through a series of empirical studies performed on standard Java classes.

This article is organized as follows. In the next Section, we start by introducing the concepts underlying our research. Next, related work is reviewed and contextualized. In Section 4, our test case generation methodology and the eCrash tool are described. The experiments conducted in order to validate and observe the impact of our proposals are discussed in Section 5, with special emphasis being put on studying the novel Test Case Evaluation and Input Domain Reduction strategies. The concluding Section presents some final considerations, the most relevant contributions, and topics for future work.

Section snippets

Background and terminology

In Evolutionary Testing, meta-heuristic search techniques are employed to select or generate test data; this section presents the most important Software Testing and Evolutionary Algorithms aspects related with this interdisciplinary area. Special attention is paid to the concepts of particular interest to our technical approach.

Related work

Evolutionary Algorithms have already been applied with significant success to the search for test data; Xanthakis et al. [47] presented the first application of heuristic optimization techniques for test data generation in 1992 [25].

However, research has been mainly geared towards generating test data for procedural software. The first approach to the field of Object-Oriented Evolutionary Testing, based on the concept of Genetic Algorithms, was presented by Tonella [39] in 2004. In this work,

Technical approach

In this section, our evolutionary approach for automatic test case generation is described. The concepts presented were implemented into the eCrash automated test case generation tool for Object-Oriented Java software [30]. The process is summarized in Fig. 3.

The eCrash tool was employed to empirically assess the impact of our Test Case Evaluation and Input Domain Reduction strategies; the experimental studies are detailed in Section 5.

Experimental studies

In this section, the empirical studies implemented with the objectives of validating and observing the impact of our Test Case Evaluation (Section 5.1) and Input Domain Reduction (Section 5.2) strategies are described and discussed.

The Java Stack and BitSet classes (JDK 1.4.2) were used as test objects. The rationale for employing these classes is related with the fact that they represent “real-world” problems and, being container classes, possess the interesting property of containing explicit

Conclusions

Evolutionary Testing is an emerging methodology for automatically generating high quality test data. It is, however, a difficult subject, especially if the aim is to implement an automated solution, viable with a reasonable amount of computational effort, which is adaptable to a wide range of test objects.

The state problem of Object-Oriented programs requires the definition of carefully fine-tuned methodologies that promote the transversal of problematic structures and difficult control-flow

Acknowledgements

This paper has been partially funded by Projects TIN2007-68083-C02 (Spanish Ministry of Education and Culture, NoHNES – Non-Hierarchical Network Evolutionary System Project), and Project GRU09105, Junta de Extremadura, Consejeria de Economia-Comercio e Innovación and FEDER.

References (49)

  • Timo Mantere et al.

    Evolutionary software engineering, a review

    Appl. Software Comput.

    (2005)
  • Andrea Arcuri, Xin Yao, A memetic algorithm for test data generation of object-oriented software, in: Proceedings of...
  • Andrea Arcuri et al.

    On test data generation of object-oriented software

  • Andrea Arcuri, Xin Yao, Search Based Testing of Containers for Object-Oriented Software, Technical Report CSR-07-3,...
  • Stéphane Barbey, Alfred Strohmeier, The problematics of testing object-oriented software, in: M. Ross, C.A. Brebbia, G....
  • Boris Beizer

    Software Testing Techniques

    (1990)
  • Antonia Bertolino

    Software testing research: achievements, challenges, dreams

  • Kenneth A. De Jong

    Evolutionary Computation

    (2002)
  • Bruce Eckel, Thinking in Java, Prentice Hall Professional Technical Reference,...
  • Mark Harman

    Automated test data generation using search based software engineering

  • Mark Harman

    The current state and future of search based software engineering

  • Mark Harman et al.

    The impact of input domain reduction on search-based test data generation

  • Thomas D. Haynes et al.

    Type inheritance in strongly typed genetic programming

  • Robert Hinterding et al.

    Adaptation in evolutionary computation: a survey

  • John H. Holland

    Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence

    (1992)
  • Kobi Inkumsah, Tao Xie, Evacon: a framework for integrating evolutionary and concolic testing for object-oriented...
  • Kobi Inkumsah, Tao Xie, Improving structural testing of object-oriented programs via integrating evolutionary testing...
  • Alex Kinneer et al.

    Sofya: supporting rapid development of dynamic program analyses for Java

  • John R. Koza

    Genetic Programming: On the Programming of Computers by Means of Natural Selection (Complex Adaptive Systems)

    (1992)
  • Gary T. Leavens et al.

    Preliminary design of jml: a behavioral interface specification language for java

    SIGSOFT Software. Eng. Notes

    (2006)
  • Kanglin Li et al.

    Effective Software Test Automation: Developing an Automated Software Testing Tool

    (2004)
  • Xiyang Liu, Bin Wang, Hehui Liu, Evolutionary search in the context of object-oriented programs, in: MIC’05:...
  • Sean Luke, ECJ 16: A Java Evolutionary Computation Library, 2007....
  • Cited by (30)

    View all citing articles on Scopus
    View full text