Stefano Tiso
ABSTRACT
Grammar-Guided Genetic Programming (GGGP) employs a variety of concepts from evolutionary theory to autonomously design solutions for a given task. Recent insights from evolutionary biology can lead to further improvements in GGGP algorithms. In this paper, we propose a new mutation approach called Facilitated Mutation (FM) that is based on the theory of Facilitated Variation. We evaluate the performance of FM on the evolution of neural network optimizers for image classification, a relevant task in Evolutionary Computation, with important implications for the field of Machine Learning. We compare FM and FM combined with crossover (FMX) against a typical mutation approach to assess the benefits of the approach. We find that FMX provides statistical improvements in key metrics, creating a superior optimizer overall (+0.5% average test accuracy), improving the average quality of solutions (+53% average population fitness), and discovering more diverse high-quality behaviors (+523 high-quality solutions discovered on average). Additionally, FM and FMX reduce the number of fitness evaluations in an evolutionary run, reducing computational costs. FM's implementation cost is minimal and the approach is theoretically applicable to any algorithm where genes are associated witha grammar non-terminal, making this approach applicable in many existing GGGP systems.
Supplemental Material
Available for Download
pdf
p558-tiso-suppl.pdf (420.7 KB)
Supplemental material.
- P. Alberch and E. A. Gale. 1985. A developmental analysis of an evolutionary trend: digital reduction in amphibians. Evolution 39, 1 (1 1985), 8--23. Google Scholar
Cross Ref
- James Bergstra and Yoshua Bengio. 2012. Random Search for Hyper-Parameter Optimization. J. Mach. Learn. Res. 13, null (feb 2012), 281--305. Google ScholarDigital Library
- P. M. Brakefield and J. C. Roskam. 2006. Exploring Evolutionary Constraints Is a Task for an Integrative Evolutionary Biology. The American Naturalist 168, S6 (2006), S4--S13. arXiv:https://doi.org/10.1086/509049 PMID: 17109328. Google ScholarCross Ref
- Mónica Campillos, Christian Von Mering, Lars Juhl Jensen, and Peer Bork. 2006. Identification and analysis of evolutionarily cohesive functional modules in protein networks. Genome Research 16, 3 (3 2006), 374--382. Google ScholarCross Ref
- Pedro Carvalho. 2023. AutoLR Facilitated Mutation Grammar. https://github.com/soren5/autolr/blob/master/grammars/adaptive_autolr_grammar_mutate_level.txtGoogle Scholar
- Pedro Carvalho. 2023. AutoLR MNIST Model. https://github.com/soren5/autolr/blob/master/models/json/mnist_model.jsonGoogle Scholar
- Pedro Carvalho. 2023. AutoLR Original Grammar. https://github.com/soren5/autolr/blob/master/grammars/adaptive_autolr_grammar_original.txtGoogle Scholar
- Pedro Carvalho, Nuno Lourenço, and Penousal Machado. 2022. Evolving Adaptive Neural Network Optimizers for Image Classification. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 13223 LNCS (2022), 3--18. Google ScholarCross Ref
- Pedro Carvalho, Jessica Mégane, Nuno Lourenço, and Penousal Machado. 2023. Context Matters: Adaptive Mutation for Grammars. In Genetic Programming: 26th European Conference, EuroGP 2023, Held as Part of EvoStar 2023, Brno, Czech Republic, April 12--14, 2023, Proceedings. Springer, 117--132.Google Scholar
- Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Yao Liu, Kaiyuan Wang, Cho-Jui Hsieh, Yifeng Lu, and Quoc V Le. 2022. Evolved Optimizer for Vision. In First Conference on Automated Machine Learning (Late-Breaking Workshop). https://openreview.net/forum?id=jK_eS5BxOuuGoogle Scholar
- Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. Routledge.Google Scholar
- Alfonso Ortega de la Puente, Rafael Sánchez Alfonso, and Manuel Alfonseca Moreno. 2002. Automatic Composition of Music by Means of Grammatical Evolution. In Proceedings of the 2002 Conference on APL: Array Processing Languages: Lore, Problems, and Applications (Madrid, Spain) (APL '02). Association for Computing Machinery, New York, NY, USA, 148--155. Google ScholarDigital Library
- Lars Feuk, Andrew R. Carson, and Stephen W. Scherer. 2006. Structural variation in the human genome. Nature Reviews Genetics 2006 7:2 7, 2 (2 2006), 85--97. Google ScholarCross Ref
- Alexander Gajewski, Kenneth O. Stanley, Jeff Clune, and Joel Lehman. 2019. Evolvability ES: Scalable and direct optimization of evolvability. In GECCO 2019 - Proceedings of the 2019 Genetic and Evolutionary Computation Conference. Association for Computing Machinery, Inc, 107--115. Google ScholarDigital Library
- John Gerhart and Marc Kirschner. 2007. The theory of facilitated variation. Proceedings of the National Academy of Sciences of the United States of America 104, SUPPL. 1 (5 2007), 8582--8589. Google ScholarCross Ref
- John Gerhart, Christopher Lowe, and Marc Kirschner. 2005. Hemichordates and the origin of chordates. Current Opinion in Genetics I& Development 15, 4 (2005), 461--467. Google ScholarCross Ref
- John C Gerhart and Marc W Kirschner. 1997. Cells, Embryos and Evolution. Blackwell Science 1997.Google Scholar
- Jens Gottlieb and Christoph Eckert. 2000. A Comparison of Two Representations for the fixed charge transportation problem. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 1917 (2000), 345--354. Google ScholarCross Ref
- Jens Gottlieb and Günther R. Raidl. 2000. Locality in decoder-based EAs for the multidimensional knapsack problem. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 1829 (2000), 38--52. Google ScholarCross Ref
- Frederic Gruau. 1994. Automatic definition of modular neural networks. Adaptive behavior 3, 2 (1994), 151--183. Google ScholarDigital Library
- Jonathan D. Gruber, Kara Vogel, Gizem Kalay, and Patricia J. Wittkopp. 2012. Contrasting properties of gene-specific regulatory, coding, and copy number mutations in saccharomyces cerevisiae: Frequency, effects, and dominance. PLoS Genetics 8, 2 (feb 2012). Google ScholarCross Ref
- Nikolaus Hansen, Dirk V Arnold, and Anne Auger. 2015. Evolution strategies. Springer handbook of computational intelligence (2015), 871--898.Google ScholarCross Ref
- Andrea Hartsock and W. James Nelson. 2008. Adherens and tight junctions: Structure, function and connections to the actin cytoskeleton. Biochimica et Biophysica Acta (BBA) - Biomembranes 1778, 3 (2008), 660--669. Google ScholarCross Ref
- Martin Hemberg, Una-May O'Reilly, Achim Menges, Katrin Jonas, Michel da Costa Gonçalves, and Steven R Fuchs. 2008. Genr8: Architects' experience with an emergent design tool. In The Art of Artificial Evolution. Springer, 167--188. Google ScholarCross Ref
- Harald Hutter, Bruce E Vogel, John D Plenefisch, Carolyn R Norris, Rui B Proenca, John Spieth, Chaobo Guo, Surjeet Mastwal, Xiaoping Zhu, Jochen Scheel, et al. 2000. Conservation and Novelty in the Evolution of Cell Adhesion and Extracellular Matrix Genes. Science 287, 5455 (2000), 989--994. Google ScholarCross Ref
- John R. Koza. 1992. Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge, MA, USA. Google ScholarCross Ref
- Michael A Lones. 2019. Instruction-level design of local optimisers using push GP. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. ACM, 1487--1494. Google ScholarDigital Library
- Michael A. Lones. 2020. Optimising Optimisers with Push GP. In Lecture Notes in Computer Science. Springer International Publishing, 101--117. Google ScholarDigital Library
- Michael A Lones. 2021. Evolving continuous optimisers from scratch. Genetic Programming and Evolvable Machines 22, 4 (2021), 395--428. Google ScholarDigital Library
- Nuno Lourenço, Francisco B. Pereira, and Ernesto Costa. 2016. Unveiling the properties of structured grammatical evolution. Genetic Programming and Evolvable Machines 17, 3 (9 2016), 251--289. Google ScholarCross Ref
- Iñigo Martincorena and Nicholas M. Luscombe. 2013. Non-random mutation: the evolution of targeted hypermutation and hypomutation. Bioessays 35, 2 (2 2013), 123--130. Google ScholarCross Ref
- Robert I McKay, Nguyen Xuan Hoai, Peter Alexander Whigham, Yin Shan, and Michael O'neill. 2010. Grammar-based genetic programming: a survey. Genetic Programming and Evolvable Machines 11, 3 (2010), 365--396. Google ScholarDigital Library
- Eric Medvet. 2017. A comparative analysis of dynamic locality and redundancy in grammatical evolution. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 10196 LNCS (2017), 326--342. Google ScholarCross Ref
- J. Grey Monroe, Thanvi Srikant, Pablo Carbonell-Bejerano, Claude Becker, Mariele Lensink, Moises Exposito-Alonso, Marie Klein, Julia Hildebrandt, Manuela Neumann, Daniel Kliebenstein, Mao-Lun Weng, Eric Imbert, Jon Ågren, Matthew T. Rutter, Charles B. Fenster, and Detlef Weigel. 2022. Mutation bias reflects natural selection in Arabidopsis thaliana. Nature 2022 (1 2022), 1--5. Google ScholarCross Ref
- Merav Parter, Nadav Kashtan, and Uri Alon. 2008. Facilitated Variation: How Evolution Learns from Past Environments To Generalize to New Environments. PLoS Computational Biology 4, 11 (2008), 1000206. Google ScholarCross Ref
- Franz Rothlauf and Marie Oetzel. 2006. On the locality of grammatical evolution. In Genetic Programming: 9th European Conference, EuroGP 2006, Budapest, Hungary, April 10--12, 2006. Proceedings 9. Springer, 320--330.Google ScholarDigital Library
- David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1985. Learning internal representations by error propagation. Technical Report. California Univ San Diego La Jolla Inst for Cognitive Science. 318--362 pages. https://ieeexplore.ieee.org/document/6302929Google Scholar
- Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. Open Ai (2017), 1--13. arXiv:1703.03864 http://arxiv.org/abs/1703.03864Google Scholar
- David L. Stern and Virginie Orgogozo. 2009. Is Genetic Evolution Predictable? Science 323, 5915 (2009), 746--751. arXiv:https://www.science.org/doi/pdf/10.1126/science.1158997 Google ScholarCross Ref
- Amos Tanay, Aviv Regev, and Ron Shamir. 2005. Conservation and evolvability in regulatory networks: The evolution of ribosomal regulation in yeast. Proceedings of the National Academy of Sciences of the United States of America 102, 20 (5 2005), 7203--7208. Google ScholarCross Ref
- Ulrich Tepass, Guy Tanentzapf, Robert Ward, and Richard Fehon. 2001. Epithelial Cell Polarity and Cell Junctions in Drosophila. Annual Review of Genetics 35, 1 (2001), 747--784. arXiv:https://doi.org/10.1146/annurev.genet.35.102401.091415 PMID: 11700298. Google ScholarCross Ref
- Tobias Uller, Armin P. Moczek, Richard A. Watson, Paul M. Brakefield, and Kevin N. Laland. 2018. Developmental bias and evolution: A regulatory network perspective. Genetics 209, 4 (2018), 949--966. Google ScholarCross Ref
- Günter P. Wagner, Mihaela Pavlicev, and James M. Cheverud. 2007. The road to modularity. Nature Reviews Genetics 8, 12 (2007), 921--931. Google ScholarCross Ref
- Richard A. Watson and Eörs Szathmáry. 2016. How Can Evolution Learn? Trends in Ecology and Evolution 31, 2 (2016), 147--157. Google ScholarCross Ref
- Carolyn A. Wessinger and Lena C. Hileman. 2016. Accessibility, constraint, and repetition in adaptive floral evolution. Developmental Biology 419, 1 (11 2016), 175--183. Google ScholarCross Ref
- Peter Alexander Whigham. 1996. Grammatical Bias for Evolutionary Learning. (1996). http://hdl.handle.net/1959.4/60693Google Scholar
- Peter A. Whigham, James MacLaurin, Grant Dick, and Caitlin A. Owen. 2015. Examining the "best of both worlds" of grammatical evolution. In GECCO 2015 - Proceedings of the 2015 Genetic and Evolutionary Computation Conference. Association for Computing Machinery, Inc, 1111--1118. Google ScholarDigital Library
- Rasmus G. Winther. 2001. Varieties of modules: Kinds, levels, origins, and behaviors. Journal of Experimental Zoology 291, 2 (8 2001), 116--129. Google ScholarCross Ref
- Gregory A. Wray. 2007. The evolutionary significance of cis-regulatory mutations. Nature Reviews Genetics 2007 8:3 8, 3 (3 2007), 206--216. Google ScholarCross Ref
- Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. Google ScholarCross Ref
Index Terms
- Biological insights on grammar-structured mutations improve fitness and diversity
Recommendations
Crossover and mutation operators for grammar-guided genetic programming
AbstractThis paper proposes a new grammar-guided genetic programming (GGGP) system by introducing two original genetic operators: crossover and mutation, which most influence the evolution process. The first, the so-called grammar-based crossover operator,...
Fitness tracking based evolutionary programming: a novel approach for function optimization
GECCO '13 Companion: Proceedings of the 15th annual conference companion on Genetic and evolutionary computationIn order to achieve a satisfactory optimization performance by evolutionary programming (EP), it is necessary to ensure proper balance between exploration and exploitation. It is obvious that one single mutation operator is not the answer. Moreover, ...
An Adaptive Genetic Algorithm Based on Population Diversity Strategy
WGEC '09: Proceedings of the 2009 Third International Conference on Genetic and Evolutionary ComputingGenetic Algorithms are adaptive methods which may be used to solve search and optimization problems. Three basic operations in Genetic Algorithms are selection, crossover and mutation, an important problem using Genetic Algorithms is the premature ...
Comments