- author = "Matheus Candido Teixeira and Gisele Lobo Pappa",
- title = "Transformers as Surrogate Models for Genetic Programming in {AutoML} Tasks",
- booktitle = "Proceedings of the 2025 Genetic and Evolutionary Computation Conference",
- year = "2025",
- editor = "Ryan Urbanowicz and Will N. Browne",
- pages = "472--480",
- address = "Malaga, Spain",
- series = "GECCO '25",
- month = "14-18 " # jul,
- organisation = "SIGEVO",
- publisher = "Association for Computing Machinery",
- publisher_address = "New York, NY, USA",
- keywords = "genetic algorithms, genetic programming, Evolutionary Machine Learning, tranformers, fitness surrogates, AutoML",
- isbn13 = "979-8-4007-1465-8",
-
URL = "
https://doi.org/10.1145/3712256.3726396",
-
DOI = "
10.1145/3712256.3726396",
- size = "9 pages",
-
abstract = "In applications where the fitness function has a high
computational cost, one of the main drawbacks of
Evolutionary Algorithms when compared to other search
methods is a prohibitive computational cost. The use of
surrogates as proxies for fitness function calculation
to alleviate this problem is not new, but addressing
the problem as a binary relation learning, i.e.,
evaluating if one individual is better or worse than
another without estimating the actual value of the
fitness, is a recent trend.
This paper proposes a transformer-encoder as a surrogate to evaluate pairs of solutions and determine their relationship, i.e., which one is better/worse than the other. We experimented the model in the context of AutoML, which seeks to find the best combination of algorithms for a classification problem. To optimize the pipeline, we can use a genetic programming, but the cost of evaluating each individual is generally expensive.
We trained the encoder with several parameters and compared its performance against traditional GP - evaluating fitness at each generation. Results confirm using the encoder as a surrogate does not degrade the fitness values of the evolved population of ML pipelines and can even improve it in some cases (up to 285 times faster).",
- notes = "GECCO-2025 EML A Recombination of the 34th International Conference on Genetic Algorithms (ICGA) and the 30th Annual Genetic Programming Conference (GP)",
