Abstract
Evolutionary ensemble learning methods with Genetic Programming have achieved remarkable results on regression and classification tasks by employing quality-diversity optimization techniques like MAP-Elites and Neuro-MAP-Elites. The MAP-Elites algorithm uses dimensionality reduction methods, such as variational auto-encoders, to reduce the high-dimensional semantic space of genetic programming to a two-dimensional behavioral space. Then, it constructs a grid of high-quality and diverse models to form an ensemble model. In MAP-Elites, however, variational auto-encoders rely on Euclidean space topology, which is not effective at preserving high-quality individuals. To solve this problem, this paper proposes a principal component analysis method based on a cosine-kernel for dimensionality reduction. In order to deal with unbalanced distributions of good individuals, we propose a zero-cost reference points synthesizing method. Experimental results on 108 datasets show that combining principal component analysis using a cosine kernel with reference points significantly improves the performance of the MAP-Elites evolutionary ensemble learning algorithm.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Balasubramanian, M., Schwartz, E.L.: The isomap algorithm and topological stability. Science 295(5552), 7–7 (2002)
Banzhaf, W., Nordin, P., Keller, R.E., Francone, F.D.: Genetic Programming: An Introduction: On the Automatic Evolution of Computer Programs and its Applications. Morgan Kaufmann Publishers Inc., Burlington (1998)
Boisvert, S., Sheppard, J.W.: Quality diversity genetic programming for learning decision tree ensembles. In: Hu, T., Lourenço, N., Medvet, E. (eds.) EuroGP 2021. LNCS, vol. 12691, pp. 3–18. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72812-0_1
Bruneton, J.P., Cazenille, L., Douin, A., Reverdy, V.: Exploration and exploitation in symbolic regression using quality-diversity and evolutionary strategies algorithms. arXiv preprint arXiv:1906.03959 (2019)
Cava, W.L., et al.: Contemporary symbolic regression methods and their relative performance. In: 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) (2021)
Cawley, G.C., Talbot, N.L.: Fast exact leave-one-out cross-validation of sparse least-squares support vector machines. Neural Netw. 17(10), 1467–1475 (2004)
Cazenille, L.: Ensemble feature extraction for multi-container quality-diversity algorithms. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 75–83 (2021)
Chen, Q., Xue, B., Zhang, M.: Preserving population diversity based on transformed semantics in genetic programming for symbolic regression. IEEE Trans. Evol. Comput. 25(3), 433–447 (2020)
Cully, A., Clune, J., Tarapore, D., Mouret, J.B.: Robots that can adapt like animals. Nature 521(7553), 503–507 (2015)
Dick, G., Owen, C.A., Whigham, P.A.: Evolving bagging ensembles using a spatially-structured niching method. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 418–425 (2018)
Dolson, E., Lalejini, A., Ofria, C.: Exploring genetic programming systems with MAP-Elites. In: Banzhaf, W., Spector, L., Sheneman, L. (eds.) Genetic Programming Theory and Practice XVI. GEC, pp. 1–16. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-04735-1_1
Gaier, A., Asteroth, A., Mouret, J.B.: Aerodynamic design exploration through surrogate-assisted illumination. In: 18th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, p. 3330 (2017)
Goldberg, D.E., Richardson, J., et al.: Genetic algorithms with sharing for multimodal function optimization. In: Genetic Algorithms and Their Applications: Proceedings of the 2nd International Conference on Genetic Algorithms, vol. 4149 (1987)
Higgins, I., et al.: beta-VAE: learning basic visual concepts with a constrained variational framework. In: International Conference on Learning Representations (2017)
Iba, H.: Bagging, boosting, and bloating in genetic programming. In: Proceedings of the 1st Annual Conference on Genetic and Evolutionary Computation-Volume 2, pp. 1053–1060 (1999)
La Cava, W., Helmuth, T., Spector, L., Moore, J.H.: A probabilistic and multi-objective analysis of lexicase selection and \(\varepsilon \)-lexicase selection. Evol. Comput. 27(3), 377–402 (2019)
La Cava, W., Singh, T.R., Taggart, J., Suri, S., Moore, J.H.: Learning concise representations for regression by evolving networks of trees. In: International Conference on Learning Representations (2018)
Lehman, J., Stanley, K.O.: Evolving a diversity of virtual creatures through novelty search and local competition. In: Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, pp. 211–218 (2011)
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: a novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. 18(1), 6765–6816 (2017)
Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. 9(11) (2008)
Moraglio, A., Krawiec, K., Johnson, C.G.: Geometric semantic genetic programming. In: Coello, C.A.C., Cutello, V., Deb, K., Forrest, S., Nicosia, G., Pavone, M. (eds.) PPSN 2012. LNCS, vol. 7491, pp. 21–31. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32937-1_3
Mouret, J.B., Clune, J.: Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909 (2015)
Ng, A., Jordan, M., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: Advances in Neural Information Processing Systems, vol. 14 (2001)
Nickerson, K., Hu, T.: Principled quality diversity for ensemble classifiers using map-elites. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 259–260 (2021)
Nickerson, K., Kolokolova, A., Hu, T.: Creating diverse ensembles for classification with genetic programming and neuro-map-elites. In: Medvet, E., Pappa, G., Xue, B. (eds.) European Conference on Genetic Programming (Part of EvoStar), pp. 212–227. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-02056-8_14
Nikfarjam, A., Neumann, A., Neumann, F.: On the use of quality diversity algorithms for the traveling thief problem. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 260–268 (2022)
Nilsson, O., Cully, A.: Policy gradient assisted map-elites. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 866–875 (2021)
Olson, R.S., La Cava, W., Orzechowski, P., Urbanowicz, R.J., Moore, J.H.: Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData Min. 10(1), 1–13 (2017)
Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Pietropolli, G., Manzoni, L., Paoletti, A., Castelli, M.: Combining geometric semantic GP with gradient-descent optimization. In: Medvet, E., Pappa, G., Xue, B. (eds.) European Conference on Genetic Programming (Part of EvoStar), pp. 19–33. Springer, Cham (2022)
Schölkopf, B., Smola, A., Müller, K.R.: Kernel principal component analysis. In: Gerstner, W., Germond, A., Hasler, M., Nicoud, J.D. (eds.) International Conference on Artificial Neural Networks, pp. 583–588. Springer, Cham (1997). https://doi.org/10.1007/BFb0020217
Urquhart, N., Hart, E.: Optimisation and illumination of a real-world workforce scheduling and routing application (WSRP) via Map-Elites. In: Auger, A., Fonseca, C.M., Lourenço, N., Machado, P., Paquete, L., Whitley, D. (eds.) PPSN 2018. LNCS, vol. 11101, pp. 488–499. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99253-2_39
Vanschoren, J., Van Rijn, J.N., Bischl, B., Torgo, L.: Openml: networked science in machine learning. ACM SIGKDD Explor. Newsl. 15(2), 49–60 (2014)
Virgolin, M.: Genetic programming is naturally suited to evolve bagging ensembles. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 830–839 (2021)
Wang, S., Mei, Y., Zhang, M.: Novel ensemble genetic programming hyper-heuristics for uncertain capacitated arc routing problem. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1093–1101 (2019)
Wang, Y., Xue, K., Qian, C.: Evolutionary diversity optimization with clustering-based selection for reinforcement learning. In: International Conference on Learning Representations (2021)
Zhang, H., Zhou, A., Zhang, H.: An evolutionary forest for regression. IEEE Trans. Evol. Comput. 26(4), 735–749 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, H., Chen, Q., Tonda, A., Xue, B., Banzhaf, W., Zhang, M. (2023). MAP-Elites with Cosine-Similarity for Evolutionary Ensemble Learning. In: Pappa, G., Giacobini, M., Vasicek, Z. (eds) Genetic Programming. EuroGP 2023. Lecture Notes in Computer Science, vol 13986. Springer, Cham. https://doi.org/10.1007/978-3-031-29573-7_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-29573-7_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-29572-0
Online ISBN: 978-3-031-29573-7
eBook Packages: Computer ScienceComputer Science (R0)