Abstract
Evolutionary optimization aims to tune the hyper-parameters during learning in a computationally fast manner. For optimization of multi-task problems, evolution is done by creating a unified search space with a dimensionality that can include all the tasks. Multi-task evolution is achieved via selective imitation where two individuals with the same type of skill are encouraged to crossover. Due to the relatedness of the tasks, the resulting offspring may have a skill for a different task. In this way, we can simultaneously evolve a population where different individuals excel in different tasks. In this paper, we consider a type of evolution called Genetic Programming (GP) where the population of genes have a tree-like structure and can be of different lengths and hence can naturally represent multiple tasks. We apply the model to multi-task neuroevolution that aims to determine the optimal hyper-parameters of a neural network such as number of nodes, learning rate, and number of training epochs using evolution. Here each gene is encoded with the hyper parameters for a single neural network. Previously, optimization was done by enabling or disabling individual connections between neurons during evolution. This method is extremely slow and does not generalize well to new neural architectures such as Seq2Seq. To overcome this limitation, we follow a modular approach where each sub-tree in a GP can be a sub-neural architecture that is preserved during crossover across multiple tasks. Lastly, in order to leverage on the inter-task covariance for faster evolutionary search, we project the features from both tasks to common space using fuzzy membership functions. The proposed model is used to determine the optimal topology of a feed-forward neural network for classification of emotions in physiological heart signals and also a Seq2seq chatbot that can converse with kindergarten children. We can outperform baselines by over 10% in accuracy.
Similar content being viewed by others
References
Cambria E, Hussain A. Sentic album: content-, concept-, and context-based online personal photo management system. Cogn Comput 2012;4:477–496.
Cambria E, Mazzocco T, Hussain A, Eckl C. Sentic medoids: organizing affective common sense knowledge in a multi-dimensional vector space. ISNN; 2011. p. 601–610.
Cambria E, Hussain A, Havasi C, Eckl C. Sentic computing: exploitation of common sense for the development of emotion-sensitive systems. Development of multimodal interfaces: active listening and synchrony; 2010. p. 148–156.
Chaturvedi I, Ong YS, Arumugam RV. Deep transfer learning for classification of time-delayed gaussian networks. Signal Process 2015;110:250–262.
Da B, Gupta A, Ong YS. Curbing negative influences online for seamless transfer evolutionary optimization. IEEE Trans Cybern 2019;49(12):4365–4378.
He T, Liu Y, Ko TH, Chan KCC, Ong YS. Contextual correlation preserving multiview featured graph clustering. IEEE Trans Cybern 2019;50(10):4318–4331.
Zhong J, Feng L, Cai W, Ong YS. Multifactorial genetic programming for symbolic regression problems. IEEE Trans Sys Man Cybern Sys 2020;50(11):4492–4505.
Li H, Ong YS, Gong M, Wang Z. Evolutionary multitasking sparse reconstruction: framework and case study. IEEE Trans Evol Comput 2019;23(5):733–747.
Oneto L, Bisio F, Cambria E, Anguita D. Semi-supervised learning for affective common-sense reasoning. Cogn Comput 2016;9:18–42.
Majumder N, Poria S, Peng H, Chhaya N, Cambria E, Gelbukh A. Sentiment and sarcasm classification with multitask learning. IEEE Intell Syst 2019;34:38–43.
Miikkulainen R. Neuroevolution. New York: Springer; 2010.
Stanley KO, Bryant BD, Miikkulainen R. Real-time neuroevolution in the nero video game. IEEE Trans Evol Comput 2005;9(6):653–668.
Li Y, Pan Q, Yang T, Wang S, Tang J, Cambria E. Learning word representations for sentiment analysis. Cogn Comput 2017;9:843–851.
Chaturvedi I, Cambria E, Cavallari S, Welsch RE. 2020. Genetic Programming for Domain Adaptation in Product Reviews, CEC. art. no. 9185713.
Bonilla EV, Chai KM, Williams C. Multi-task gaussian process prediction. NIPS. In: Platt JC, Koller D, Singer Y, and Roweis ST, editors; 2008. p. 153–160.
Ong YS, Gupta A. Evolutionary multitasking: a computer science view of cognitive multitasking. Cogn Comput 2016;8(2):125–142.
Feng L, Zhou L, Zhong J, Gupta A, Ong YS, Tan K, Qin AK. Evolutionary multitasking via explicit autoencoding. IEEE Trans Cybern 2019;49(9):3457–3470.
Snoek J, Larochelle H, Adams RP. Practical Bayesian optimiza tion of machine learning algorithms. NIPS; 2012. p. 2960–2968.
Richards N, Moriarty DE, Miikkulainen R. Evolving neural networks to play go. Appl Intel 1998;8(1):85–96.
Suganuma M, Shirakawa S, Nagao T. A genetic programming approach to designing convolutional neural network architectures. IJCAI; 2018. p. 5369–5373.
Yao X, Islam MM. Evolving artificial neural network ensembles. IEEE Comput Intel Mag 2008; 3(1):31–42.
Fernandes E, De Carvalho AC, Yao X. Ensemble of classifiers based on multiobjective genetic sampling for imbalanced data. IEEE Trans Knowl Data Eng 2020;32(6):1104–1115.
Susanto Y, Livingstone AG, Ng BC, Cambria E. The hourglass model revisited. IEEE Intelligent Systems 2020;35(5):96–102.
Chaturvedi I, Satapathy R, Cavallari S, Cambria E. Fuzzy commonsense reasoning for multimodal sentiment analysis. Pattern Recogn Lett 2019;125:264–270.
Chung H, Iorga M, Voas J, Lee S. Alexa, can i trust you? Computer 2017;50(9):100–104.
Ghazvininejad M, Brockett C, Chang M-W, Dolan B, Gao J, tau Yih W, Galley M. A knowledge-grounded neural conversation model. AAAI; 2018. p. 5110–5117.
Cambria E, Li Y, Xing FZ, Poria S, Kwok K. 2020. SenticNet 6: Ensemble Application of Symbolic and Subsymbolic AI for Sentiment Analysis. CIKM, pp. 105–114.
Chen C-Y, Yu D, Wen W, Yang YM, Zhang J, Zhou M, Jesse K, Chau A, Bhowmick A, Iyer S, Sreenivasulu G, Cheng R, Bhandare A, Yu Z. 2018. Gunrock: building a human-like social bot by leveraging large scale real user data.
Young T, Cambria E, Chaturvedi I, Huang M, Zhou H, Biswas S. Augmenting end-to-end dialog systems with commonsense knowledge. AAAI; 2018. p. 4970–4977.
Chaturvedi I, Ragusa E, Gastaldo P, Cambria E. 2020. COAL: Convolutional Online Adaptation Learning for Opinion Mining, ICDM Workshops.
Katsigiannis S, Ramzan N. Dreamer: a database for emotion recognition through eeg and ecg signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Info 2018;22(1):98–107.
Miranda Correa JA, Abadi MK, Sebe N, Patras I. 2018. Amigos: a dataset for affect, personality and mood research on individuals and groups. IEEE Trans Affect Comput: 1–1.
Subramanian R, Wache J, Abadi MK, Vieriu RL, Winkler S, Sebe N. Ascertain: emotion and personality recognition using commercial sensors. IEEE Trans Affect Comput 2018;9(2):147–160.
Golberg H, Paradis J, Crago M. Lexical acquisition over time in minority first language children learning english as a second language. Applied Psycholinguistics 2008;29(1):41–65.
Funding
This work is internally funded by the College of Science and Engineering at Jame Cook University. This work is also partially supported by the Data Science and Artificial Intelligence Center (DSAIR) at the Nanyang Technological University.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Additional information
Ethical Approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Chaturvedi, I., Su, C.L. & Welsch, R.E. Fuzzy Aggregated Topology Evolution for Cognitive Multi-tasks. Cogn Comput 13, 96–107 (2021). https://doi.org/10.1007/s12559-020-09807-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12559-020-09807-4