- author = "Alexander Edward Ian Brownlee and James Callan and Karine Even-Mendoza and Alina Geiger and Carol Hanna and Justyna Petke and Federica Sarro and Dominik Sobania",
- title = "Large Language Model Based Mutations in Genetic Improvement",
- journal = "Automated Software Engineering",
- year = "2024",
- note = "Accepted, to appear",
- note = "Special Issue on Advances in Search-Based Software",
- keywords = "genetic algorithms, genetic programming, Genetic Improvement, SBSE, GIN, LLM, AI, ANN, JavaParser, JVM, stochastic local search",
-
URL = "
https://doi.org/10.21203/rs.3.rs-4437272/v1",
-
data_url = "https://zenodo.org/records/11173088",
- size = "20 pages",
- abstract = "we evaluate the use of LLMs as mutation operators for genetic improvement (GI), an SBSE approach, to improve the GI search process. In a preliminary work, we explored the feasibility of combining the Gin Java GI toolkit with OpenAI LLMs in order to generate an edit for the JCodec tool. Here we extend this investigation involving three LLMs and three types of prompt, and five real-world software projects. We sample the edits at random, as well as using local search. Our results show that, compared with conventional statement GI edits, LLMs produce fewer unique edits, but these compile and pass tests more often, with the OpenAI model finding test-passing edits 77percent of the time. The OpenAI and Mistral LLMs were roughly equal in finding the best run-time improvements. Simpler prompts were more successful than those providing more context and examples. Qualitative analysis revealed a wide variety of areas where LLMs typically failed to produce valid edits: commonly including inconsistent formatting, generating non-Java syntax, or refusing to provide a solution",
-
notes = "Extends
\cite{Brownlee:2023:SSBSE}
ThreadMXBean.getThreadCPUTime, Langchain4J, TinyDolphin, Mistral, small changes prompt, structural changes prompt.
JCodec, JUnit4, Gson, Commons-Net, Karate. Profiling. Maven 3.9.x.
Sect 5.1 'over 800 valid patches were found'
Sect 8 'up to 9 times more [than with random] patches passing the unit tests were found with LLM-based edits.' 'the prompts used for LLM requests greatly affect the results'
",
