Abstract
Designing a Recurrent Neural Network to extract sentiment from tweets is a very hard task. When using memory cells in their design, the task becomes even harder due to the large number of design alternatives and the costly process of finding a performant design. In this paper we propose an original evolutionary algorithm to address the hard challenge of discovering novel Recurrent Neural Network memory cell designs for sentiment analysis on tweets. We used three different tasks to discover and evaluate the designs. We conducted experiments and the results show that the best obtained designs surpass the baselines—which are the most popular cells, LSTM and GRU. During the discovery process we evaluated roughly 17,000 cell designs. The selected winning candidate outperformed the others for the overall sentiment analysis problem, hence showing generality. We made the winner selection by using the cumulated accuracies on all three considered tasks.
Similar content being viewed by others
References
L. Araujo, Genetic programming for natural language processing. Genet. Program. Evol. Mach. 21(1), 11–32 (2020)
T. Bäck, H.P. Schwefel, An overview of evolutionary algorithms for parameter optimization. Evol. Comput. 1(1), 1–23 (1993)
B. Baker, O. Gupta, R. Raskar, N. Naik, Accelerating neural architecture search using performance prediction. arXiv preprint arXiv:1705.10823 (2017)
J. Bayer, D. Wierstra, J. Togelius, J. Schmidhuber, Evolving memory cell structures for sequence learning. in International Conference on Artificial Neural Networks (Springer, 2009), pp. 755–764
M. Becker, M. Staniek, V. Nastase, A. Palmer, A. Frank, Classifying semantic clause types: modeling context and genre characteristics with recurrent neural networks and attention. in Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017) (2017), pp. 230–240
R. Breuer, R. Kimmel, A deep learning perspective on the origin of facial expressions. arXiv preprint arXiv:1705.01842 (2017)
M.Y. Chen, H.S. Chiang, A.K. Sangaiah, T.C. Hsieh, Recurrent neural network with attention mechanism for language model. in Neural Computing and Applications (2019), pp. 1–9
K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
T. Chugh, K. Sindhya, J. Hakanen, K. Miettinen, A survey on handling computationally expensive multiobjective optimization problems with evolutionary algorithms. Soft Comput. 23(9), 3137–3166 (2019)
A.M. Ciobanu, L.P. Dinu, Automatic identification and production of related words for historical linguistics. Comput. Linguist. 667–704 (2012)
K. Deb, Multi-objective Optimization Using Evolutionary Algorithms, vol. 16 (Wiley, Hoboken, 2001)
S. Ding, H. Li, C. Su, J. Yu, F. Jin, Evolutionary artificial neural networks: a review. Artif. Intell. Rev. 39(3), 251–260 (2013)
P. Doetsch, M. Kozielski, H. Ney, Fast and robust training of recurrent neural networks for offline handwriting recognition. in 2014 14th International Conference on Frontiers in Handwriting Recognition, pp. 279–284. (IEEE, 2014)
C.A. Duchanoy, M.A. Moreno-Armendáriz, L. Urbina, C.A. Cruz-Villar, H. Calvo, J.d.J. Rubio, A novel recurrent neural network soft sensor via a differential evolution training algorithm for the tire contact patch. Neurocomputing 235, 71–82 (2017)
F.A. Gers, J. Schmidhuber, Recurrent nets that time and count. in Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, vol. 3 (IEEE, 2000), pp. 189–194
F. Gers, J. Schmidhuber, F. Cummins, Learning to forget: continual prediction with LSTM. Neural Comput. 12, 2451–2471 (2000). https://doi.org/10.1162/089976600300015015
F. Gomez, R. Miikkulainen, Incremental evolution of complex general behavior. Adapt. Behav. 5(3–4), 317–342 (1997)
K. Greff, R.K. Srivastava, J. Koutník, B.R. Steunebrink, J. Schmidhuber, LSTM: A search space odyssey. IEEE Trans. Neural Netw. Learn. Syst. 28(10), 2222–2232 (2016)
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 770–778
S. Hochreiter, J. Schmidhuber, Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
T.J. Hsieh, H.F. Hsiao, W.C. Yeh, Forecasting stock markets using wavelet transforms and recurrent neural networks: An integrated system based on artificial bee colony algorithm. Appl. Soft Comput. 11(2), 2510–2525 (2011)
E. Kouloumpis, T. Wilson, J. Moore, Twitter sentiment analysis: the good the bad and the omg! in Fifth International AAAI Conference on Weblogs and Social Media (2011)
T. Lei, Y. Zhang, S.I. Wang, H. Dai, Y. Artzi, Simple recurrent units for highly parallelizable recurrence. arXiv preprint arXiv:1709.02755 (2017)
Z.C. Lipton, J. Berkowitz, C. Elkan, A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv:1506.00019 (2015)
D.C. Mallqui, R.A. Fernandes, Predicting the direction, maximum, minimum and closing prices of daily bitcoin exchange rate using machine learning techniques. Appl. Soft Comput. 75, 596–606 (2019)
S.M. Mohammad, F. Bravo-Marquez, Emotion intensities in tweets. arXiv preprint arXiv:1708.03696 (2017)
C.I. Muntean, G.A. Morar, D. Moldovan, Exploring the meaning behind twitter hashtags through clustering. in International Conference on Business Information Systems. (Springer, 2012), pp. 231–242
I. Naji, Twitter Sentiment Analysis Training Corpus (Dataset). http://thinknook.com/twitter-sentiment-analysis-training-corpus-dataset-2012-09-22/ (2012). Accessed 21 July 2020
S. Otte, M. Liwicki, A. Zell, Dynamic cortex memory: enhancing recurrent neural networks for gradient-based sequence learning. in International Conference on Artificial Neural Networks (Springer, 2014), pp. 1–8
A. Pak, P. Paroubek, Twitter as a corpus for sentiment analysis and opinion mining. LREc 10, 1320–1326 (2010)
A. Rawal, R. Miikkulainen, From nodes to networks: Evolving recurrent neural networks. arXiv preprint arXiv:1803.04439 (2018)
H. Sak, A. Senior, F. Beaufays, Long short-term memory recurrent neural network architectures for large scale acoustic modeling. in: Fifteenth Annual Conference of the International Speech Communication Association (2014)
R. Sprugnoli, S. Tonelli, Novel event detection and classification for historical texts. Comput. Linguist. 45(2), 229–265 (2019)
G. Stilo, P. Velardi, Hashtag sense clustering based on temporal similarity. Comput. Linguist. 43(1), 181–200 (2017)
D. Stojanovski, G. Strezoski, G. Madjarov, I. Dimitrovski, Finki at semeval-2016 task 4: Deep learning architecture for twitter sentiment analysis. in Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) (2016), pp. 149–154
M. Suganuma, S. Shirakawa, T. Nagao, A genetic programming approach to designing convolutional neural network architectures. in Proceedings of the Genetic and Evolutionary Computation Conference (2017), pp. 497–504
I. Sutskever, O. Vinyals, Q. Le, Sequence to sequence learning with neural networks. Advances in NIPS (2014)
J. Wehrmann, W. Becker, H.E. Cagnini, R.C. Barros, A character-based convolutional neural network for language-agnostic twitter sentiment analysis. in 2017 International Joint Conference on Neural Networks (IJCNN). (IEEE, 2017), pp. 2384–2391
P.J. Werbos, Backpropagation through time: what it does and how to do it. Proc. IEEE 78(10), 1550–1560 (1990)
S. Xingjian, Z. Chen, H. Wang, D.Y. Yeung, W.K. Wong, W.c. Woo: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. in Advances in Neural Information Processing Systems (2015), pp. 802–810
X. Yao, Evolving artificial neural networks. Proc. IEEE 87(9), 1423–1447 (1999)
E. Zitzler, Evolutionary Algorithms for Multiobjective Optimization: Methods and Applications, vol. 63. (Citeseer, 1999)
B. Zoph, Q.V. Le, Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The handling Associate Editor "Sebastian Risi" is a Neural Systems editor for this paper.
Rights and permissions
About this article
Cite this article
Nistor, S.C., Moca, M. & Nistor, R.L. Discovering novel memory cell designs for sentiment analysis on tweets. Genet Program Evolvable Mach 22, 147–187 (2021). https://doi.org/10.1007/s10710-020-09395-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10710-020-09395-0