CGenProg: Adaptation of cartesian genetic programming with migration and opposite guesses for automatic repair of software regression faults

https://doi.org/10.1016/j.eswa.2020.114503Get rights and content

Highlights

  • We propose CGenProg for automatic repair of software regression faults in Java programs.

  • We adapted and modified cartesian genetic programming as the core evolutionary algorithm.

  • We adapted biogeography-based optimization (migration) as the crossover.

  • We adapted opposition-based learning (opposite guesses) as the mutation.

Abstract

In the last decade, the research community has been actively working to develop the techniques that can automatically find a solution to a software fault, namely, automatic program repair (APR). As of today, a multitude of APR techniques has been proposed. The techniques could have effectively repaired a wide variety of fault classes. However, the development of effective APR techniques for software regression faults, which are prevalently occurred in the maintenance stage of the software lifecycle, have received little attention. By incorporating specific knowledge in the domain of software regression faults, we have developed a novel technique for automatic repair of software regression faults in Java programs, which we call CGenProg. To achieve this, we have extensively adapted and modified the original cartesian genetic programming (CGP), biogeography-based optimization (BBO), and opposition-based learning (OBL). The modified CGP serves us as the core evolutionary process while the modified BBO and OBL act as crossover and mutation, respectively. The significance of CGenProg is that it contributes to the solution of a practical problem faced by developers in the maintenance stage of the software lifecycle. For expert and intelligent systems, it extends what is known about the application of optimization algorithms in the context of APR. Further, it demonstrates a novel use of CGP, BBO, and OBL for automatic repair of software regression faults. To evaluate CGenProg, we have developed a prototype tool using the Java language. Then, we conducted experiments on several programs in Code4Bench where each program is released with multiple consecutive versions comprising software regression faults. In the experiments, CGenProg could repair 17 out of 30 faulty programs. We conclude that CGenProg proves relevant and effective for repairing software regression faults. The impact of this study is to incentivize researchers for further exploitation and adaptation of the wealth of existing metaheuristics to develop effective APR techniques for different fault classes.

Introduction

Many tasks today are carried out with the software. Hence, the software must necessarily be reliable. However, faults are still prevalent in the software code (Monperrus, 2018), which make debugging an indispensable activity of the software lifecycle. Since manual debugging is labor-intensive and can result in the introduction of new faults, the researchers have commenced to develop automatic program repair (APR) techniques. APR involves a broad family of techniques (Gazzola et al., 2019, Monperrus, 2018) that attempt to find a solution for the software faults automatically. Consequently, they can alleviate the developer’s burden, to reduce the debugging costs, and to increase the quality of the software (Assiri & Bieman, 2017). APR can be considered as a subfield of automatic programming, in which the latter’s goal is to develop a computer program from scratch. Among many metaheuristics, automatic programming has extensively exploited from genetic programming. For instance, the well-known, baseline APR technique, GenProg (Le Goues, Nguyen, Forrest, & Weimer, 2012), has been designed based on an adapted variant of genetic programming. GenProg has led to impressive results and undergone several improvements (Fast et al., 2010, Le Goues et al., 2012, Oliveira et al., 2016). Genetic programming has also been applied to improve functional and non-functional requirements of the software (Langdon & Harman, 2015). To enhance the performance and efficiency of genetic programming, numerous extensions have been proposed, of which cartesian genetic programming (CGP) (Miller, 2019, Yazdani and Shanbehzadeh, 2015) is notable. CGP has been used to solve a wide spectrum of automatic programming problems (Olmo, Romero, & Ventura, 2014), such as automatic generation of digital circuits, medical diagnosis (Miller, 2019a), solving symbolic regression equations (Yazdani & Shanbehzadeh, 2015), feature extraction (Yazdani, Shanbehzadeh, & Hadavandi, 2017), and genetic improvement of non-functional requirement of software (Mrazek, Vasicek, & Sekanina, 2015). However, it has not been used for the APR problem yet (Miller, 2019b, Manazir and Raza, 2019).

Recently, Tan and Roychoudhury (2015) proposed RELIFIX for automatic repair of regression faults in C programs. These faults are introduced in the current version of the software due to the modifications to the previous version and can cause the software to regress. The importance of resolving regression faults can be evidenced in the development of a plethora of mature techniques for regression testing (Yoo and Harman, 2012, Khatibsyarbini et al., 2018, Souto and d’Amorim, 2018, Andrews et al., 2019, Ali et al., 2020, Mondal and Nasre, 2019, Minhas et al., 2020). Despite the efficacy in the experiments, relifix is subject to a couple of issues. First, the repair operators are manually extracted. Second, the operators have been extracted from a specific, limited dataset. These issues can lead to the reduction in the degree of automation and overfitting of the fixed code to the repair operators, respectively. Aimed at resolving the issues, this paper concerns itself to close the debugging loop of regression faults through the presentation of a new APR technique.

Inspired from GenProg, we have extensively modified the original CGP with respect to representation and evolutionary operators to develop a novel technique, namely CGenProg, which is short for Cartesian Genetic Programming for automatic program repair. To design CGenProg, we have leveraged some insights and knowledge from the regression faults domain.

CGenProg is a generate-and-validate (Le Goues, Forrest, & Weimer, 2013), syntax-based (Gazzola et al., 2019) APR technique in the realm of search-based software engineering (Harman, Mansouri, & Zhang, 2012), which can be used to automatically repair real-world regression faults in Java programs. It receives as input, a faulty version of software, along with the last correct version, and a test suite. The test suite must include passing test cases that are required to encode the core (required) functionality. In addition, the test suite must include at least a failing test case that encodes the fault1. As output, CGenProg generates a patch, which is an evolved program that passes on whole test cases of the given test suite2. Several of CGenProg features, particularly those that are pertinent to its requirements, share commonalities with GenProg. For instance, test cases can be taken from regression test suite, generated by a developer or an automatic technique, or can be the steps of a fault reproduction. Additionally, CGenProg is independent of any formal specifications or annotations. Furthermore, it can be applied on whole source code or an individual module.

Fig. 1 depicts the high-level process of CGenProg. As genotype structure, we have developed a novel representation, which is taken from the original CGP genotype. To determine the likelihood to which each statement of the input program is faulty, we apply a fault localization technique, which has been developed based on dynamic information of Jaccard (Zou, Liang, Xiong, Ernst, & Zhang, 2019) combined with some static information. To evolve the individuals, crossover and mutations are employed. We developed a novel crossover based on statement migration, which is inspired from biogeography-based optimization (BBO) (Simon, 2008). In addition, we developed a novel mutation operator based on the idea of opposite guesses, which is also inspired from opposition-based learning (OBL) (Ergezer, Simon, & Du, 2009). The process of CGenProg searches for a software version that retains the core functionality and fixes the fault in question. CGenProg works generally at statement-level and partially at expression-level. The representation and evolutionary operators of CGenProg are highly modified, extended, and adapted for automatic repair of regression faults. During the development of CGenProg, we have exploited the lessons and knowledge from APRSuite (Khalilian, Baraani-Dastjerdi, & Zamani, 2019) to justify our hypotheses and design decisions.

In order to evaluate CGenProg, we have developed a prototype tool and conducted empirical studies on several Java subject programs taken from Code4Bench (Majd, Vahidi-Asl, Khalilian, Baraani-Dastjerdi, & Zamani, 2019). Each subject program comprises consecutive versions, some of which involve software regression faults. In the experiments, CGenProg could successfully repair 17 out of 30 faulty versions. The obtained results corroborate that CGenProg is effective in repairing software regression faults. The significance of this study is that it contributes to the solution of a widespread practical problem that is frequently faced by developers. In addition, the impact of this study is to incentivize researchers to exploit the rich source of existing metaheuristics and optimization algorithms in the APR context.

The main contributions of this paper are as follows:

  • extensive adaptation and modification of CGP, BBO, and OBL for the APR context

  • CGenProg, a novel APR technique customized for the specific domain of software regression faults

  • a prototype tool that implements CGenProg

  • empirical studies to evaluate the effectiveness of CGenProg along with quantitative and qualitative analysis of the results

  • an objective, systematic, and qualitative comparison of CGenProg with two baseline APR techniques

  • a detailed discussion on various considerations, implications, computational and practical aspects, and trade-offs regarding the design decisions of CGenProg and its implementation

The remainder of this paper is organized as follows: In Section 2, we explain the required concepts and the motivation to our own work. In Section 3, we elaborate on the details of CGenProg, its components, design decisions, and the underlying hypotheses. In Section 4, we present the details of our empirical studies. In Section 5, we provide discussions on the CGenProg itself and the evaluation results. We provide related work in Section 6. Finally, we conclude the paper in Section 7 and explain future work.

Section snippets

Concepts and motivation

In this section, we discuss around some necessary concepts and illuminate the domain of regression faults, which is the specific domain and class of the faults that our technique is intended to repair. Then, we explain the issues with the current regression repair techniques and highlight the challenges of fixing regression faults. Next, we present our problem statement. Finally, we leverage an illustrative example by which we delineate the challenges and draw several questions in algorithmic

The proposed technique

In this section, we explain the details of our new technique for automatic repair of software regression faults, namely, CGenProg. First, we discuss around the reasons behind the usage of CGP as the core evolutionary algorithm of CGenProg. Then, we introduce the high-level structure of CGenProg. In the subsequent sections, we give the details of different constituent components of CGenProg.

Empirical studies

In this section, we explain the details of how we have evaluated CGenProg. To achieve this, we present our implementation, experimental subjects, and the obtained results.

Discussion

In this section, we discuss around consequences of CGenProg from different dimensions. First, we investigate some of the considerations, high-level insights, trade-offs, and the implications regarding the conception and design decisions. Then, we explain several concerns, limitations, and threats to the validity associated with the implementation and evaluation of CGenProg.

Related work

A myriad of studies in different subfields of software engineering and evolutionary computation are relevant to our study. In this section, we review the major body of the literature that are closely related to our own study. We have grouped the related work in several categories with each category devoted to a select set of studies.

The Domain of Regression Faults. Software regression faults are frequently occurred during the maintenance stage of the software lifecycle. Hence, detecting

Conclusions and future work

The goal of this study was to develop an APR technique that can be effectively used to repair software regression faults. We accomplished this goal by introducing CGenProg that exploits the specific knowledge of software regression faults. We have extensively modified and adapted CGP, BBO, and OBL to develop CGenProg in which our design decisions and hypotheses have been taken and justified based on the current APR knowledge. In CGenProg, CGP acts as the core evolutionary algorithm while BBO

CRediT authorship contribution statement

Alireza Khalilian: Conceptualization. Ahmad Baraani-Dastjerdi: Supervision. Bahman Zamani: Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

We are grateful to the anonymous reviewers of the journal. Their comments have led to considerable improvements to our paper.

References (126)

  • X. Kong et al.

    The impacts of techniques, programs and tests on automated program repair: An empirical study

    Journal of Systems and Software

    (2018)
  • L.-L. Li et al.

    Biogeography-based optimization based on population competition strategy for solving the substation location problem

    Expert Systems with Applications

    (2018)
  • X. Ma et al.

    MOEA/D with opposition-based learning for multiobjective optimization problem

    Neurocomputing

    (2014)
  • S. Mahdavi et al.

    Opposition based learning: A literature review

    Swarm and Evolutionary Computation

    (2018)
  • A. Majd et al.

    Code4Bench: A multidimensional benchmark of Codeforces data for different program analysis techniques

    Journal of Computer Languages

    (2019)
  • N.M. Minhas et al.

    Regression testing for large-scale embedded software development – Exploring the state of practice

    Information and Software Technology

    (2020)
  • S. Mondal et al.

    Mahtab: Phase-wise acceleration of regression testing for C

    Journal of Systems and Software

    (2019)
  • M.A. Remli et al.

    An enhanced scatter search with combined opposition-based learning for parameter estimation in large-scale kinetic models of biochemical systems

    Engineering Applications of Artificial Intelligence

    (2017)
  • A.P. Rifai et al.

    Non-dominated sorting biogeography-based optimization for bi-objective reentrant flexible manufacturing system scheduling

    Applied Soft Computing

    (2018)
  • R. Sarkhel et al.

    An improved Harmony Search Algorithm embedded with a novel piecewise opposition based learning algorithm

    Engineering Applications of Artificial Intelligence

    (2018)
  • R. Abou Assi et al.

    Coincidental correctness in the Defects4J benchmark

    Software Testing, Verification and Reliability

    (2019)
  • S. Ali et al.

    Enhanced regression testing technique for agile software development and continuous integration strategies

    Software Quality Journal

    (2020)
  • Aho, A. V., Lam. S. M., Sethi, R., Ullman, J. D. (2006). Principles of compiler design. Reading:...
  • P. Ammann et al.

    Introduction to software testing

    (2016)
  • Arcuri, A. (2009). Automatic software generation and improvement through search based techniques (Doctoral...
  • A. Arcuri et al.

    A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering

    Software Testing, Verification and Reliability

    (2014)
  • F.Y. Assiri et al.

    Fault localization for automated program repair: Effectiveness, performance, repair correctness

    Software Quality Journal

    (2017)
  • Barr, E. T., Brun, Y., Devanbu, P., Harman, M., & Sarro, F. (2014). The plastic surgery hypothesis. InProceedings of...
  • T.V. Belle et al.
  • Brun, Y., Barr, E., Xiao, M., Le Goues, C., & Devanbu, P. (2013). Evolution vs. intelligent design in program...
  • Burke, E. D., & Kendall, G. (2014). Search methodologies: introductory tutorials in optimization and decision support...
  • Cordy, M., Rwemalika, R., Papadakis, M., & Harman, M. (2019). FlakiMe: Laboratory-Controlled Test Flakiness Impact...
  • E.G. Daylight et al.

    The dawn of software engineering: From turing to Dijkstra

    Lonely Scholar.

    (2012)
  • Do, H. (2016). Recent advances in regression testing techniques. In Advances in Computers (Vol. 103, pp. 53-77)....
  • S.S. Emam et al.

    Test case prioritization using extended digraphs

    ACM Transactions on Software Engineering and Methodology

    (2015)
  • M. Ergezer et al.
  • Ernst, M. D., Cockrell, J., Griswold, W. G., & Notkin, D. (2001). Dynamically discovering likely program invariants to...
  • Fast, E., Le Goues, C., Forrest, S., & Weimer, W. (2010). Designing better fitness functions for automated program...
  • Fry, Z. P., Landau, B., & Weimer, W. (2012). A human study of patch maintainability. InProceedings of the 2012...
  • Gabel, M., & Su, Z. (2010). A study of the uniqueness of source code. InProceedings of the eighteenth ACM SIGSOFT...
  • L. Gazzola et al.

    Automatic software repair: A survey

    IEEE Transactions on Software Engineering

    (2019)
  • Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization and Machine Learning....
  • Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT...
  • Harman, M. (2010). Technical Perspective Automated Patching Techniques: The Fix Is In.Communications of the...
  • M. Harman et al.

    Search-based software engineering: Trends, techniques and applications

    ACM Computing Surveys

    (2012)
  • Harper, R. (2012). Spatial co-evolution: quicker, fitter and less bloated. In Proceedings of the 14th annual conference...
  • A. Hindle et al.
  • J.H. Holland

    Genetic Algorithms

    Scientific American

    (1992)
  • Holland, J. H. (1992b). Adaptation in natural and artificial systems: an introductory analysis with applications to...
  • J.N. Hooker

    Testing heuristics: We have it all wrong

    J Heuristics

    (1995)
  • Cited by (0)

    View full text