CGenProg: Adaptation of cartesian genetic programming with migration and opposite guesses for automatic repair of software regression faults
Introduction
Many tasks today are carried out with the software. Hence, the software must necessarily be reliable. However, faults are still prevalent in the software code (Monperrus, 2018), which make debugging an indispensable activity of the software lifecycle. Since manual debugging is labor-intensive and can result in the introduction of new faults, the researchers have commenced to develop automatic program repair (APR) techniques. APR involves a broad family of techniques (Gazzola et al., 2019, Monperrus, 2018) that attempt to find a solution for the software faults automatically. Consequently, they can alleviate the developer’s burden, to reduce the debugging costs, and to increase the quality of the software (Assiri & Bieman, 2017). APR can be considered as a subfield of automatic programming, in which the latter’s goal is to develop a computer program from scratch. Among many metaheuristics, automatic programming has extensively exploited from genetic programming. For instance, the well-known, baseline APR technique, GenProg (Le Goues, Nguyen, Forrest, & Weimer, 2012), has been designed based on an adapted variant of genetic programming. GenProg has led to impressive results and undergone several improvements (Fast et al., 2010, Le Goues et al., 2012, Oliveira et al., 2016). Genetic programming has also been applied to improve functional and non-functional requirements of the software (Langdon & Harman, 2015). To enhance the performance and efficiency of genetic programming, numerous extensions have been proposed, of which cartesian genetic programming (CGP) (Miller, 2019, Yazdani and Shanbehzadeh, 2015) is notable. CGP has been used to solve a wide spectrum of automatic programming problems (Olmo, Romero, & Ventura, 2014), such as automatic generation of digital circuits, medical diagnosis (Miller, 2019a), solving symbolic regression equations (Yazdani & Shanbehzadeh, 2015), feature extraction (Yazdani, Shanbehzadeh, & Hadavandi, 2017), and genetic improvement of non-functional requirement of software (Mrazek, Vasicek, & Sekanina, 2015). However, it has not been used for the APR problem yet (Miller, 2019b, Manazir and Raza, 2019).
Recently, Tan and Roychoudhury (2015) proposed RELIFIX for automatic repair of regression faults in C programs. These faults are introduced in the current version of the software due to the modifications to the previous version and can cause the software to regress. The importance of resolving regression faults can be evidenced in the development of a plethora of mature techniques for regression testing (Yoo and Harman, 2012, Khatibsyarbini et al., 2018, Souto and d’Amorim, 2018, Andrews et al., 2019, Ali et al., 2020, Mondal and Nasre, 2019, Minhas et al., 2020). Despite the efficacy in the experiments, relifix is subject to a couple of issues. First, the repair operators are manually extracted. Second, the operators have been extracted from a specific, limited dataset. These issues can lead to the reduction in the degree of automation and overfitting of the fixed code to the repair operators, respectively. Aimed at resolving the issues, this paper concerns itself to close the debugging loop of regression faults through the presentation of a new APR technique.
Inspired from GenProg, we have extensively modified the original CGP with respect to representation and evolutionary operators to develop a novel technique, namely CGenProg, which is short for Cartesian Genetic Programming for automatic program repair. To design CGenProg, we have leveraged some insights and knowledge from the regression faults domain.
CGenProg is a generate-and-validate (Le Goues, Forrest, & Weimer, 2013), syntax-based (Gazzola et al., 2019) APR technique in the realm of search-based software engineering (Harman, Mansouri, & Zhang, 2012), which can be used to automatically repair real-world regression faults in Java programs. It receives as input, a faulty version of software, along with the last correct version, and a test suite. The test suite must include passing test cases that are required to encode the core (required) functionality. In addition, the test suite must include at least a failing test case that encodes the fault1. As output, CGenProg generates a patch, which is an evolved program that passes on whole test cases of the given test suite2. Several of CGenProg features, particularly those that are pertinent to its requirements, share commonalities with GenProg. For instance, test cases can be taken from regression test suite, generated by a developer or an automatic technique, or can be the steps of a fault reproduction. Additionally, CGenProg is independent of any formal specifications or annotations. Furthermore, it can be applied on whole source code or an individual module.
Fig. 1 depicts the high-level process of CGenProg. As genotype structure, we have developed a novel representation, which is taken from the original CGP genotype. To determine the likelihood to which each statement of the input program is faulty, we apply a fault localization technique, which has been developed based on dynamic information of Jaccard (Zou, Liang, Xiong, Ernst, & Zhang, 2019) combined with some static information. To evolve the individuals, crossover and mutations are employed. We developed a novel crossover based on statement migration, which is inspired from biogeography-based optimization (BBO) (Simon, 2008). In addition, we developed a novel mutation operator based on the idea of opposite guesses, which is also inspired from opposition-based learning (OBL) (Ergezer, Simon, & Du, 2009). The process of CGenProg searches for a software version that retains the core functionality and fixes the fault in question. CGenProg works generally at statement-level and partially at expression-level. The representation and evolutionary operators of CGenProg are highly modified, extended, and adapted for automatic repair of regression faults. During the development of CGenProg, we have exploited the lessons and knowledge from APRSuite (Khalilian, Baraani-Dastjerdi, & Zamani, 2019) to justify our hypotheses and design decisions.
In order to evaluate CGenProg, we have developed a prototype tool and conducted empirical studies on several Java subject programs taken from Code4Bench (Majd, Vahidi-Asl, Khalilian, Baraani-Dastjerdi, & Zamani, 2019). Each subject program comprises consecutive versions, some of which involve software regression faults. In the experiments, CGenProg could successfully repair 17 out of 30 faulty versions. The obtained results corroborate that CGenProg is effective in repairing software regression faults. The significance of this study is that it contributes to the solution of a widespread practical problem that is frequently faced by developers. In addition, the impact of this study is to incentivize researchers to exploit the rich source of existing metaheuristics and optimization algorithms in the APR context.
The main contributions of this paper are as follows:
- •
extensive adaptation and modification of CGP, BBO, and OBL for the APR context
- •
CGenProg, a novel APR technique customized for the specific domain of software regression faults
- •
a prototype tool that implements CGenProg
- •
empirical studies to evaluate the effectiveness of CGenProg along with quantitative and qualitative analysis of the results
- •
an objective, systematic, and qualitative comparison of CGenProg with two baseline APR techniques
- •
a detailed discussion on various considerations, implications, computational and practical aspects, and trade-offs regarding the design decisions of CGenProg and its implementation
The remainder of this paper is organized as follows: In Section 2, we explain the required concepts and the motivation to our own work. In Section 3, we elaborate on the details of CGenProg, its components, design decisions, and the underlying hypotheses. In Section 4, we present the details of our empirical studies. In Section 5, we provide discussions on the CGenProg itself and the evaluation results. We provide related work in Section 6. Finally, we conclude the paper in Section 7 and explain future work.
Section snippets
Concepts and motivation
In this section, we discuss around some necessary concepts and illuminate the domain of regression faults, which is the specific domain and class of the faults that our technique is intended to repair. Then, we explain the issues with the current regression repair techniques and highlight the challenges of fixing regression faults. Next, we present our problem statement. Finally, we leverage an illustrative example by which we delineate the challenges and draw several questions in algorithmic
The proposed technique
In this section, we explain the details of our new technique for automatic repair of software regression faults, namely, CGenProg. First, we discuss around the reasons behind the usage of CGP as the core evolutionary algorithm of CGenProg. Then, we introduce the high-level structure of CGenProg. In the subsequent sections, we give the details of different constituent components of CGenProg.
Empirical studies
In this section, we explain the details of how we have evaluated CGenProg. To achieve this, we present our implementation, experimental subjects, and the obtained results.
Discussion
In this section, we discuss around consequences of CGenProg from different dimensions. First, we investigate some of the considerations, high-level insights, trade-offs, and the implications regarding the conception and design decisions. Then, we explain several concerns, limitations, and threats to the validity associated with the implementation and evaluation of CGenProg.
Related work
A myriad of studies in different subfields of software engineering and evolutionary computation are relevant to our study. In this section, we review the major body of the literature that are closely related to our own study. We have grouped the related work in several categories with each category devoted to a select set of studies.
The Domain of Regression Faults. Software regression faults are frequently occurred during the maintenance stage of the software lifecycle. Hence, detecting
Conclusions and future work
The goal of this study was to develop an APR technique that can be effectively used to repair software regression faults. We accomplished this goal by introducing CGenProg that exploits the specific knowledge of software regression faults. We have extensively modified and adapted CGP, BBO, and OBL to develop CGenProg in which our design decisions and hypotheses have been taken and justified based on the current APR knowledge. In CGenProg, CGP acts as the core evolutionary algorithm while BBO
CRediT authorship contribution statement
Alireza Khalilian: Conceptualization. Ahmad Baraani-Dastjerdi: Supervision. Bahman Zamani: Supervision.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgement
We are grateful to the anonymous reviewers of the journal. Their comments have led to considerable improvements to our paper.
References (126)
- et al.
Opposition-based learning in shuffled frog leaping: An application for parameter identification
Information Sciences
(2015) - et al.
New feature selection methods based on opposition-based learning and self-adaptive cohort intelligence for predicting patient no-shows
Applied Soft Computing
(2020) - et al.
Black-box model-based regression testing of fail-safe behavior in web applications
Journal of Systems and Software
(2019) Evolutionary repair of faulty software
Applied Soft Computing
(2011)- et al.
Co-evolutionary automatic programming for software development
Information Sciences
(2014) Quasi-oppositional differential evolution for optimal reactive power dispatch
International Journal of Electrical Power & Energy Systems
(2016)- et al.
The Daikon system for dynamic detection of likely invariants
Science of Computer Programming
(2007) - et al.
A hybrid self-adaptive sine cosine algorithm with opposition based learning
Expert Systems with Applications
(2019) - et al.
APRSuite: A suite of components and use cases based on categorical decomposition of automatic program repair techniques and tools
Journal of Computer Languages
(2020) - et al.
Test case prioritization approaches in regression testing: A systematic literature review
Information and Software Technology
(2018)