Prediction of faults-slip-through in large software projects: an empirical evaluation

Abstract

A large percentage of the cost of rework can be avoided by finding more faults earlier in a software test process. Therefore, determination of which software test phases to focus improvement work on has considerable industrial interest. We evaluate a number of prediction techniques for predicting the number of faults slipping through to unit, function, integration, and system test phases of a large industrial project. The objective is to quantify improvement potential in different test phases by striving toward finding the faults in the right phase. The results show that a range of techniques are found to be useful in predicting the number of faults slipping through to the four test phases; however, the group of search-based techniques (genetic programming, gene expression programming, artificial immune recognition system, and particle swarm optimization–based artificial neural network) consistently give better predictions, having a representation at all of the test phases. Human predictions are consistently better at two of the four test phases. We conclude that the human predictions regarding the number of faults slipping through to various test phases can be well supported by the use of search-based techniques. A combination of human and an automated search mechanism (such as any of the search-based techniques) has the potential to provide improved prediction results.

Appendix: Model training and testing procedure

This section discusses the parameter settings that have been considered for different techniques during model selection. These settings may be used for a future replication of this study and to quantify the impact of changing the parameter settings, perhaps using different data sets. As given in Sect. 3.1, we use data from 45 weeks of the baseline project to train the models, while the results are evaluated on the data from 15 weeks of an ongoing project. The experimental evaluation process is also summarized in Procedure 1.

The least-square multiple regression does not require selection of parameters, rather the coefficients are determined from the training data. Different estimators implemented in the WEKA machine-learning tool (Hall et al. 2009) have been evaluated for pace regression that includes empirical Bayes, ordinary least square, Akaike’s information criterion (AIC), and risk inflation criterion (RIC). The estimator giving the least ARE is selected as the best pace regression model.

The M5P technique requires setting the minimum number of instances at a leaf node and has been varied in the range [2, 4, …, 10] with pruning and smoothing. The model with minimum ARE is retained. The REPTree technique requires setting the maximum depth of the tree, the minimum total weight of the instances in a leaf, the minimum variance proportion at a node required for splitting, the number of folds of data used for pruning, and the seed value used for randomizing the data. We have imposed no restriction on the maximum depth of the tree while the minimum total weight of the instances in a leaf is varied in the range [2,4, …, 10]. The minimum variance proportion at a node, the number of folds of data used for pruning, and the seed value used for randomization are kept constant at their default values of 0.0010, 3, and 1, respectively.

The K* instance-based technique requires setting the blending parameter that has a value between 0 and 100 %. This parameter has been varied in the range of [0, 20, 40, …, 100]. For k-NN, the number of neighbors has been varied in the range of [1, 3, 5, …, 15].

For SVM, two types of parameters have to be set by the user, i.e., values for the epsilon parameter, $\varepsilon$ and the regularization parameter, C . Setting the value of C near the range of the output values has been found to be a successful heuristic. We therefore vary C within the range [1, 3, …, 11]. The value of $\varepsilon$ is varied in the range [0.001, 0.003] while the kernel used is the radial basis function. Training an artificial neural network (ANN) requires deciding on the number of layers and the number of nodes at each layer. We considered the ANN architecture with 1 input layer, 2 hidden layers, and 1 output layer. The number of independent variables in the problem determined the number of input nodes. The two hidden layers used a varied number of nodes in the range [1, 3, 5, 7], while the output layer used a single node. The hyperbolic tangent sigmoid and linear transfer functions have been used for the hidden and output nodes, respectively. Finally, the number of epochs used is 500 and the weights are updated using a learning rate of 0.3 and a momentum of 0.2.

Model selection for Bagging involves deciding upon the size of the bag as a percentage of the training set size and the number of iterations to be performed. These two parameters have been varied in the range [25, 50, 75, 100] and [5, 10, 15], respectively. The REPTree technique is used as the base learner. For rotation forest, the number of iterations have been varied in the range [5, 10, 15] and the base learner used is the REPTree technique.

GP requires setting a number of control parameters. Although the affect of changing these control parameters on the end solution is still an active area of research, we nevertheless experimented with different function and terminal sets. Initially, we experimented with a minimal set of functions and the terminal set containing the independent variable only. We incrementally increased the function set with additional functions and later on also complemented the terminal set with a random constant. The best model having the best fitness was chosen from all the runs of the GP system with different variations of function and terminal sets. The GP programs were evaluated according to the sum of absolute differences between the obtained and expected results in all fitness cases, $\sum\nolimits_{i=1}^n |e_{i}-e_{i}^{\prime}|,$ where e _i is the actual fault count data, $e_i^{\prime}$ is the estimated value of the fault count data and n is the size of the data set used to train the GP models. The control parameters that were chosen for the GP system are shown in Table 9. For GEP, the solutions are evaluated for fitness using mean squared error and the control parameters are shown in Table 10. The AIRS algorithm also requires setting a number of parameters. While it is not possible to experiment with all the different combinations of these parameters, however, the value of k for the majority voting has been varied in the range [1, 3, 5, …, 15]. Rest of the parameters used were as follows: affinity threshold = 0.2, clonal rate = 10, hypermutation rate = 2, mutation rate = 0.1, stimulation value = 0.9, and total resources = 150. For PSO-ANN, the architecture similar to the basic ANN is followed except that the weights are now optimized using PSO with the number of particles in the swarm set to 25 and the number of iterations varied in the range [500, 1,000, 15,000, 2,000]. The mean squared error is used as the fitness function.

Table 9 GP control parameters

Table 10 GEP control parameters