PSB2: The Second Program Synthesis Benchmark Suite

PSB2: The Second Program Synthesis Benchmark Suite Datasets

Version 1.0.1 (see version history at bottom)

This repository contains datasets for the 25 problems described in the paper PSB2: The Second Program Synthesis Benchmark Suite. These problems come from a variety of sources, and require a range of programming constructs and datatypes to solve. These datasets are designed to be usable for any method of performing general program synthesis, including and not limited to inductive program synthesis and evolutionary methods such as genetic programming.

For more information, see the associated website: https://cs.hamilton.edu/~thelmuth/PSB2/PSB2.html

Use

Each problem in the benchmark suite is located in a separate directory in the `datasets` directory.

For each problem, we provide a set of `edge` cases and a set of `random` cases. The `edge` cases are hand-chosen cases representing the limits of the problem. The `random` cases are all generated based on problem-specific distributions. For each problem, we included exactly 1 million `random` cases.

A typical use of these datasets for a set of runs of program synthesis would be:

• For each run, use every `edge` case in the training set
• For each run, use a different, randomly-sampled set of `random` cases in the training set.
• Use a larger set of `random` cases as an unseen test set.

Sampling Libraries

We provide the following libraries to make the downloading and sampling of these datasets easier. Using these libraries, you do not need to download the entire dataset from Zenodo; the individual problem datasets are downloaded and stored once when first sampling them.

• Python: https://github.com/thelmuth/psb2-python
• Clojure: https://github.com/thelmuth/psb2-clojure

Dataset format

Each edge and random dataset is provided in three formats: CSV, JSON, and EDN, with all three formats containing identical data.

The CSV files are formatted as follows:

• The first row of the file is the column names.
• Each following row corresponds to one set of program inputs and expected outputs.
• Input columns are labeled `input1`, `input2`, etc., and output columns are labeled `output1`, `output2`, etc.
• In CSVs, string inputs and outputs are double quoted when necessary, but not if not necessary. Newlines within strings are escaped.
• Columns in CSV files are comma-separated.

The JSON and EDN files are formatted using the JSON Lines standard (adapted for EDN).
Each case is put on its own line of the data file. The files should be read line-by-line and each parsed into an object/map using a JSON/EDN parser.

Citation

If you use these datasets in a publication, please cite the paper PSB2: The Second Program Synthesis Benchmark Suite and include a link to this repository.

BibTeX entry for paper:

@InProceedings{Helmuth:2021:GECCO,
  author =	"Thomas Helmuth and Peter Kelly",
  title =	"{PSB2}: The Second Program Synthesis Benchmark Suite",
  booktitle =	"2021 Genetic and Evolutionary Computation Conference",
  series = {GECCO '21},
  year = 	"2021",
  isbn13 = {978-1-4503-8350-9},
  address = {Lille, France},
  size = {10 pages},
  doi = {10.1145/3449639.3459285},
  publisher = {ACM},
  publisher_address = {New York, NY, USA},
  month = {10-14} # jul,
  doi-url = {https://doi.org/10.1145/3449639.3459285},
  URL = {https://dl.acm.org/doi/10.1145/3449639.3459285},
}

Version History

1.0.0 - 2021/4/10 - Initial publication of PSB2 datasets on Zenodo.

1.0.1 - 2021/7/9 - Changes to CSVs to quote all strings that could be read as integers. No changes in actual data, just formatting.