Exploring True Test Overfitting in Dynamic Automated Program Repair using Formal Methods

Created by W.Langdon from gp-bibliography.bib Revision:1.8081

@InProceedings{DBLP:conf/icst/NilizadehLLPC21,

• author = "Amirfarhad Nilizadeh and Gary T. Leavens and Xuan-Bach Dinh Le and Corina S. Pasareanu and David R. Cok",
• title = "Exploring True Test Overfitting in Dynamic Automated Program Repair using Formal Methods",
• booktitle = "14th IEEE Conference on Software Testing, Verification and Validation, ICST 2021",
• year = "2021",
• pages = "229--240",
• address = "Porto de Galinhas, Brazil",
• month = apr # " 12-16",
• keywords = "genetic algorithms, genetic programming, GenProg, Automated Program Repair, APR, test overfitting, formal methods",
• timestamp = "Fri, 22 Oct 2021 09:07:13 +0200",
• biburl = "https://dblp.org/rec/conf/icst/NilizadehLLPC21.bib",
• bibsource = "dblp computer science bibliography, https://dblp.org",
• URL = "https://doi.org/10.1109/ICST49551.2021.00033",
• DOI = "doi:10.1109/ICST49551.2021.00033",
• URL = "https://program-repair.org/bibliography.html",
• code_url = "https://github.com/Amirfarhad-Nilizadeh/Java-JML",
• size = "12 pages",
• abstract = "Automated program repair (APR) techniques have shown a promising ability to generate patches that fix program bugs automatically. Typically such APR tools are dynamic in the sense that they find bugs by testing and they validate patches by running a program's test suite. Patches can also be validated manually. However, neither of these methods for validating patches can truly tell whether a patch is correct. Test suites are usually incomplete, and thus APR-generated patches may pass the tests but not be truly correct; in other words, the APR tools may be overfitting to the tests. The possibility of test overfitting leads to manual validation, which is costly, potentially biased, and can also be incomplete. Therefore, we must move past these methods to truly assess APR overfitting problem.

  We aim to evaluate the test overfitting problem in dynamic APR tools using ground truth given by a set of programs equipped with formal behavioral specifications. Using these formal specifications and an automated verification tool, we found that there is definitely overfitting in the generated patches of seven well-studied APR tools, although many (about 59percent) of the generated patches were indeed correct. Our study further points out two new problems that can affect APR tools: changes to the complexity of programs and numeric problems. An additional contribution is that we introduce the first publicly available data set of formally specified and verified Java programs, their test suites, and buggy variants, each of which has exactly one bug.",

• notes = "'Our study uses seven dynamic APR tools for Java:' Cardumen, jGenProg, jKali, jMutRepair, ARJAE, Kali-A, Nopol.

  also known as \cite{NilizICTS21}, \cite{9438573}

  University of Central Florida, Orlando, Florida, USA",

}

Genetic Programming entries for Amirfarhad Nilizadeh Gary T Leavens Xuan-Bach Dinh Le Corina S Pasareanu David R Cok

Citations