Test Case Evaluation and Input Domain Reduction strategies for the Evolutionary Testing of Object-Oriented software
Introduction
Software testing is expensive, typically consuming roughly half of the total costs involved in software development while adding nothing to the raw functionality of the final product. Yet, it remains the primary method through which confidence in software is achieved [6]. A large amount of the resources spent on testing are applied on the difficult and time consuming task of locating quality test data; automating this process is vital to advance the state-of-the-art in software testing. However, automation in this area has been quite limited, mainly because the exhaustive enumeration of a program’s input is unfeasible for any reasonably-sized program, and random methods are unlikely to exercise “deeper” features of software [25].
Meta-heuristic search techniques, like Evolutionary Algorithms – high-level frameworks which utilise heuristics, inspired by genetics and natural selection, in order to find solutions to combinatorial problems at a reasonable computational cost [4] – are natural candidates to address this problem, since the input space is typically large but well defined, and test goal can usually be expressed as a fitness function [10].
The application of Evolutionary Algorithms to test data generation is often referred to as Evolutionary Testing [39] or Search-Based Testing [25]. Approaches have been proposed that focus on the usage of Genetic Algorithms [16], [17], [39], [48], Ant Colony Optimization [22], Genetic Programming [38], Strongly-Typed Genetic Programming [43], [45], and Memetic Algorithms [1].
Evolutionary Testing is an emerging methodology for automatically generating high quality test data. It is, however, a difficult subject, especially if the aim is to implement an automated solution, viable with a reasonable amount of computational effort, which is adaptable to a wide range of test objects. Significant success has been achieved by applying this technique to the automatic generation of unit-test cases for procedural software [24], [25]. The application of search-based strategies for Object-Oriented unit-testing is, however, fairly recent [39] and is yet to be investigated comprehensively [11].
The focus of our research is precisely on developing a solution for employing Evolutionary Algorithms for generating test sets for the structural unit-testing of Object-Oriented programs. Our approach involves representing and evolving test cases using the Strongly-Typed Genetic Programming technique [28]. The methodology for evaluating the quality of test cases includes instrumenting the program under test, and executing it using the generated test cases as inputs with the intention of collecting trace information with which to derive coverage metrics. The aim is that of efficiently guiding the search process towards achieving full structural coverage of the program under test. These concepts have been implemented into the eCrash automated test case generation tool – which will be described below.
Our main goals are those of defining strategies for addressing the challenges posed by the Object-Oriented paradigm and of proposing methodologies for enhancing the efficiency of search-based testing approaches. The primary contributions of this work are the following:
- •
Presenting a strategy for Test Case Evaluation and search guidance, which involves allowing unfeasible test cases (i.e., those that terminate prematurely due to a runtime exception) to be considered at certain stages of the evolutionary search – namely, once the feasible test cases that are being bred cease to be interesting.
- •
Introducing a novel Input Domain Reduction methodology, based on the concept of Purity Analysis, which allows the identification and removal of entries that are irrelevant to the search problem because they do not contribute to the definition of test scenarios.
Additionally, our methodology for automated test case generation is thoroughly described and validated through a series of empirical studies performed on standard Java classes.
This article is organized as follows. In the next Section, we start by introducing the concepts underlying our research. Next, related work is reviewed and contextualized. In Section 4, our test case generation methodology and the eCrash tool are described. The experiments conducted in order to validate and observe the impact of our proposals are discussed in Section 5, with special emphasis being put on studying the novel Test Case Evaluation and Input Domain Reduction strategies. The concluding Section presents some final considerations, the most relevant contributions, and topics for future work.
Section snippets
Background and terminology
In Evolutionary Testing, meta-heuristic search techniques are employed to select or generate test data; this section presents the most important Software Testing and Evolutionary Algorithms aspects related with this interdisciplinary area. Special attention is paid to the concepts of particular interest to our technical approach.
Related work
Evolutionary Algorithms have already been applied with significant success to the search for test data; Xanthakis et al. [47] presented the first application of heuristic optimization techniques for test data generation in 1992 [25].
However, research has been mainly geared towards generating test data for procedural software. The first approach to the field of Object-Oriented Evolutionary Testing, based on the concept of Genetic Algorithms, was presented by Tonella [39] in 2004. In this work,
Technical approach
In this section, our evolutionary approach for automatic test case generation is described. The concepts presented were implemented into the eCrash automated test case generation tool for Object-Oriented Java software [30]. The process is summarized in Fig. 3.
The eCrash tool was employed to empirically assess the impact of our Test Case Evaluation and Input Domain Reduction strategies; the experimental studies are detailed in Section 5.
Experimental studies
In this section, the empirical studies implemented with the objectives of validating and observing the impact of our Test Case Evaluation (Section 5.1) and Input Domain Reduction (Section 5.2) strategies are described and discussed.
The Java Stack and BitSet classes (JDK 1.4.2) were used as test objects. The rationale for employing these classes is related with the fact that they represent “real-world” problems and, being container classes, possess the interesting property of containing explicit
Conclusions
Evolutionary Testing is an emerging methodology for automatically generating high quality test data. It is, however, a difficult subject, especially if the aim is to implement an automated solution, viable with a reasonable amount of computational effort, which is adaptable to a wide range of test objects.
The state problem of Object-Oriented programs requires the definition of carefully fine-tuned methodologies that promote the transversal of problematic structures and difficult control-flow
Acknowledgements
This paper has been partially funded by Projects TIN2007-68083-C02 (Spanish Ministry of Education and Culture, NoHNES – Non-Hierarchical Network Evolutionary System Project), and Project GRU09105, Junta de Extremadura, Consejeria de Economia-Comercio e Innovación and FEDER.
References (49)
- et al.
Evolutionary software engineering, a review
Appl. Software Comput.
(2005) - Andrea Arcuri, Xin Yao, A memetic algorithm for test data generation of object-oriented software, in: Proceedings of...
- et al.
On test data generation of object-oriented software
- Andrea Arcuri, Xin Yao, Search Based Testing of Containers for Object-Oriented Software, Technical Report CSR-07-3,...
- Stéphane Barbey, Alfred Strohmeier, The problematics of testing object-oriented software, in: M. Ross, C.A. Brebbia, G....
Software Testing Techniques
(1990)Software testing research: achievements, challenges, dreams
Evolutionary Computation
(2002)- Bruce Eckel, Thinking in Java, Prentice Hall Professional Technical Reference,...
Automated test data generation using search based software engineering
The current state and future of search based software engineering
The impact of input domain reduction on search-based test data generation
Type inheritance in strongly typed genetic programming
Adaptation in evolutionary computation: a survey
Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence
Sofya: supporting rapid development of dynamic program analyses for Java
Genetic Programming: On the Programming of Computers by Means of Natural Selection (Complex Adaptive Systems)
Preliminary design of jml: a behavioral interface specification language for java
SIGSOFT Software. Eng. Notes
Effective Software Test Automation: Developing an Automated Software Testing Tool
Cited by (30)
Nature-inspired Approaches in Software Faults Identification and Debugging
2016, Procedia Computer ScienceAutomatic assessment of object oriented programming assignments with unit testing in Python and a real case assignment
2023, Computer Applications in Engineering EducationEvolutionary Approaches to Test Data Generation for Object-Oriented Software: Overview of Techniques and Tools
2021, Research Anthology on Agile Software, Software Development, and TestingInput domain reduction of search-based structural test data generation using interval arithmetic
2018, International Journal of Performability Engineering