- author = "Thomas Helmuth and Peter Kelly",
- title = "Applying genetic programming to {PSB2}: the next generation program synthesis benchmark suite",
- journal = "Genetic Programming and Evolvable Machines",
- year = "2022",
- volume = "23",
- number = "3",
- pages = "375--404",
- month = sep,
- note = "Special Issue: Highlights of Genetic Programming 2021 Events",
- keywords = "genetic algorithms, genetic programming, Automatic program synthesis, Benchmarking, PushGP",
- ISSN = "1389-2576",
-
DOI = "
doi:10.1007/s10710-022-09434-y",
-
code_url = "
https://cs.hamilton.edu/%7ethelmuth/PSB2/PSB2.html",
-
code_url = "https://github.com/thelmuth/Clojush/releases/tag/psb2-v1.0",
-
code_url = "https://zenodo.org/record/5084812#.YxCloPfTVmM",
- size = "30 pages",
- abstract = "For the past seven years, researchers in genetic programming and other program synthesis disciplines have used the General Program Synthesis Benchmark Suite (PSB1) to benchmark many aspects of systems that conduct programming by example, where the specifications of the desired program are given as input/output pairs. PSB1 has been used to make notable progress toward the goal of general program synthesis: automatically creating the types of software that human programmers code. Many of the systems that have attempted the problems in PSB1 have used it to demonstrate performance improvements granted through new techniques. Over time, the suite has gradually become outdated, hindering the accurate measurement of further improvements. The field needs a new set of more difficult benchmark problems to move beyond what was previously possible and ensure that systems do not overfit to one benchmark suite. In this paper, we describe the 25 new general program synthesis benchmark problems that make up PSB2, a new benchmark suite. These problems are curated from a variety of sources, including programming katas and college courses. We selected these problems to be more difficult than those in the original suite, and give results using PushGP showing this increase in difficulty. We additionally give an example of benchmarking using a state-of-the-art parent selection method, showing improved performance on PSB2 while still leaving plenty of room for improvement. These new problems will help guide program synthesis research for years to come.",
-
notes = "Extends \cite{Helmuth:2021:GECCO} PSB1 =
\cite{Helmuth:2015:GECCO}
'General program synthesis requires the manipulation of multiple data types' '4 multi-output problems in PSB2.' 'large training sets (200 examples) that have a variety of specific edge cases purposefully included.' 'we did not include any problems on which standard PushGP produced a success rate over 60 percent in initial experiments'.
'We aimed to include problems that require a large variety of data types and control flow structures to solve, with a balance between data types across problems. Most of the problems require some type of iteration and/or conditional statements. Required data types include integers, floats, Booleans, characters, strings, vectors of integers, and vectors of floats. In order to produce large datasets, we aimed to select problems that have at least 1 million possible unique inputs.'",
