- author = "Mingyi Lim and Giovani Guizzo and Justyna Petke",
- title = "Impact of Test Suite Coverage on Overfitting in Genetic Improvement of Software",
- booktitle = "12th International Symposium on Search Based Software Engineering SSBSE 2020",
- year = "2020",
- editor = "Juan Pablo Galeotti and Bonita Sharif",
- series = "LNCS",
- volume = "12420",
- pages = "188--203",
- address = "Bari, Italy",
- month = "7-8 " # oct,
- publisher = "Springer",
- keywords = "genetic algorithms, genetic programming, Genetic Improvement, SBSE, GIN, EvoSuite",
- isbn13 = "978-3-030-59761-0",
-
URL = "
http://www.cs.ucl.ac.uk/staff/J.Petke/papers/Lim_2020_SSBSE_RENE.pdf",
-
video_url = "
https://www.youtube.com/watch?v=018oszfebqg",
-
DOI = "
doi:10.1007/978-3-030-59762-7_14",
- size = "16 pages",
- abstract = "Genetic Improvement (GI) uses automated search to improve existing software. It can be used to improve runtime, energy consumption, fix bugs, and any other software property, provided that such property can be encoded into a fitness function. GI usually relies on testing to check whether the changes disrupt the intended functionality of the software, which makes test suites important artefacts for the overall success of GI. The objective of this work is to establish which characteristics of the test suites correlate with the effectiveness of GI. We hypothesise that different test suite properties may have different levels of correlation to the ratio between overfitting and non-overfitting patches generated by the GI algorithm. In order to test our hypothesis, we perform a set of experiments with automatically generated test suites using EvoSuite and 4 popular coverage criteria. We used these test suites as input to a GI process and collected the patches generated throughout such a process. We find that while test suite coverage has an impact on the ability of GI to produce correct patches, with branch coverage leading to least overfitting, the overfitting rate was still significant. We also compared automatically generated tests with manual, developer-written ones and found that while manual tests had lower coverage, the GI runs with manual tests led to less overfitting than in the case of automatically generated tests. Finally, we did not observe enough statistically significant correlations between the coverage metrics and overfitting ratios of patches, i.e., the coverage of test suites cannot be used as a linear predictor for the level of overfitting of the generated patches.",
-
notes = "
Replications and Negative Results
line, branch, conditional branch, and weak mutation coverage (ie how many mutants does the test suite detect).
Replication package code https://github.com/justynapt/ssbse2020RENE python3, java, R, MacOS
GI & EvoSuite to reduce runtime of Java programs. output diversity metric. Triangle and nine sort programs, 14 to 52 lines of Java code. Gin PatchAnalyser held out test suite.
420000 patches generated and tested in 16 hours.
'GI can indeed find valid and non-overfitting patches'
'manually created test suites generate patches that overfit significantly less than patches generated with automatically generated test suites.'
With EvoSuite 'test suite (coverage) measures seem to have no bearing on how good a test suite is for the purpose of applying genetic improvement'
http://ssbse2020.di.uniba.it/accepted-papers/",
