Victor Caetano
ABSTRACT
Representation learning is an area responsible for learning data representations that makes it easier for machine learning algorithms to extract useful information from them. Deep learning currently has the most effective methods for this task and can learn distributed representations - also known as embeddings - able to represent different properties of the data and their relationship. In this direction, this paper introduces a new way to look at tree-like GP individuals for symbolic regression. Given a set of predefined operators and a sufficiently large number of solutions sampled from the space, we train a transformer to learn an encoding/decoding function. By transforming a tree representation into a distributed representation, we are able to measure distances between trees in a much more efficient way and, more importantly, generate the potential for these representations to capture semantics. We show the distance accounting for embedding presents results very similar to those of a tree-edition, which reflects their syntactic similarity. Although the model as it stands is not able to capture semantics yet, we show its potential by using the generated tree-representation model in a simple task: measuring distances between trees in a fitness-sharing scenario.
- Bengio, Y.: Deep learning of representations: Looking forward. In: International conference on statistical language and speech processing. pp. 1--37. Springer (2013)Google Scholar
Digital Library
- Burke, E.K., Gustafson, S., Kendall, G.: Diversity in Genetic Programming: An Analysis of Measures and Correlation with Fitness. IEEE Trans. Evol. Comput. 8(1), 47--62 (2004)Google ScholarDigital Library
- Bäck, T., Fogel, D., Michalewicz, Z.: Evolutionary Computation 2---Advanced Algorithms and Operators (01 2000). Google ScholarCross Ref
- Chen, Q., Xue, B., Zhang, M.: Preserving Population Diversity Based on Transformed Semantics in Genetic Programming for Symbolic Regression. IEEE Trans. Evol. Comput. 25(3), 433--447 (2021)Google ScholarDigital Library
- Demšar, J.: Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research 7, 1--30 (2006)Google Scholar
- Kenton, J.D.M.W.C., Toutanova, L.K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT. pp. 4171--4186 (2019)Google Scholar
- Liu, Z., Lin, Y., Sun, M.: Representation learning for natural language processing. Springer Nature (2020)Google Scholar
- McInnes, L., Healy, J., Melville, J.: Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)Google Scholar
- Mikolov, T., Corrado, G., Chen, K., Dean, J.: Efficient Estimation of Word Representations in Vector Space. In: ICLR. pp. 1--12 (2013)Google Scholar
- Olson, R.S., La Cava, W., Orzechowski, P., Urbanowicz, R.J., Moore, J.H.: Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData Mining 10(1), 36 (Dec 2017)Google ScholarCross Ref
- Orzechowski, P., La Cava, W., Moore, J.H.: Where are we now? a large benchmark study of recent symbolic regression methods. In: Proceedings of the Genetic and Evolutionary Computation Conference. p. 1183--1190. GECCO '18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3205455.3205539 Google ScholarDigital Library
- Shiv, V.L., Quirk, C.: Novel positional encodings to enable tree-based transformers. In: NeurIPS. pp. 12058--12068 (2019)Google Scholar
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. arXiv preprint arXiv:1706.03762 (2017)Google Scholar
- Zhang, K., Shasha, D.: Simple fast algorithms for the editing distance between trees and related problems. SIAM journal on computing 18(6), 1245--1262 (1989)Google Scholar
Index Terms
- Symbolic Regression Trees as Embedded Representations
Recommendations
Universal Succinct Representations of Trees?
ICALP '09: Proceedings of the 36th International Colloquium on Automata, Languages and Programming: Part IWe consider the succinct representation of ordinal and cardinal trees on the RAM with logarithmic word size. Given a tree T , our representations support the following operations in O (1) time: (i) $\mbox{{\tt BP-substring}}(i,b)$, which reports the ...
Transformation-interaction-rational representation for symbolic regression
GECCO '22: Proceedings of the Genetic and Evolutionary Computation ConferenceSymbolic Regression searches for a function form that approximates a dataset often using Genetic Programming. Since there is usually no restriction to what form the function can have, Genetic Programming may return a hard to understand model due to non-...
Quantifying the space-efficiency of 2D graphical representations of trees
A mathematical evaluation and comparison of the space-efficiency of various 2D graphical representations of tree structures is presented. As part of the evaluation, a novel metric called the mean area exponent is introduced that quantifies the ...
Comments